Image processing apparatus

ABSTRACT

An image processing apparatus, in which an object image focused by a lens is split into a plurality of images by means of a light splitting section. These images are converted into image data items by a plurality of imaging devices which are arranged with their imaging area overlapping in part. The image data items are stored temporarily in an image storing section. A displacement detecting section detects displacement coefficients (rotation angle R and parallel displacement S) from the image signals representing the mutual overlap region of two images which are to be combined and which are represented by two image data items read from the image storing section. The position of any specified pixel of the image displayed is identified by the pixel signal generated by the corresponding pixel of any imaging device. An interpolation section performs interpolation on the pixel values of the imaging device, thereby correcting the values of the other pixels of the image displayed and ultimately generating interpolated image signals. The interpolated image signals are combined with the image signals produced by the imaging device, whereby a display section displays a high-resolution image.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This is a continuation of U.S. patent application Ser. No.09/740,764 filed Dec. 19, 2000, which is a divisional application ofU.S. patent application Ser. No. 08/969,937 filed Nov. 28, 1999, whichis a continuation of U.S. patent application Ser. No. 08/045,038 filedApr. 8, 1993, the entire contents of each of which are incorporatedherein by reference.

[0002] This application is based upon and claims the benefit of priorityunder 35 USC 119 of prior Japanese Patent Applications No. 4-089090,filed Apr. 9, 1992; No. 4-239803 filed Sep. 8, 1992; and No. 5-042402filed Mar. 3, 1993.

[0003] 1. Field of the Invention

[0004] The present invention relates to an image processing apparatusfor forming either images of the parts of an object or images of anobject which are identical but different in color, and for combining theimages into a wide high-resolution image of the object.

[0005] 2. Description of the Related Art

[0006] Image processing apparatuses using a solid-state imaging devicesuch as a CCD are generally used in electronic still cameras, videocameras, and the like. It is demanded that an image processing apparatushave a higher resolution, particularly so high a resolution that theapparatus may provide a wide image of an object. Also it is desired thatthe image processing apparatus have so high a resolution that it canform an image as wide as a panoramic image.

[0007] Two techniques are available for increasing the resolution of theimage processing apparatus. The first technique is to use a solid-stateimaging device with a sufficiently high resolution. The second techniqueis to use a plurality of solid-state imaging devices for obtainingimages of parts of an object, respectively, and to combine the imagesinto a single high-resolution image of the entire object.

[0008] More precisely, the first resolution-increasing technique is touse more pixels per unit area of the device chip. In other words,smaller pixels are arranged in a greater number in the unit area, thusincreasing the pixel density of the imaging device.

[0009] The second resolution-increasing technique is classified into twotypes. The first-type technique com incorporated in an image processingapparatus, thereby switching the view field of the apparatus from onepart of an object to another part and thus enabling the imaging devicesto produce images of parts of an object, and the second step ofcombining the images, thus produced, into a high-resolution image of theentire object. The second-type technique comprises the first step ofdividing an optical image 600 of an object into, for example, four partsby means of prisms as shown in FIG. 1, the second step of applying theparts of the optical image to four imaging devices 611, 621, 631, and641, respectively, and the third step of combining the image data itemsoutput by the devices, thereby forming a single image of the object. Inthe second-type technique, the imaging devices 611 to 641 are sopositioned as to cover the predetermined parts of the object asillustrated in FIG. 2.

[0010] There is known another resolution-increasing technique similar tothe second-type technique described in the preceding paragraph. Thistechnique uses a detector 611 having four imaging devices 612 which arearranged in the same plane in a 2×2 matrix, spaced apart from oneanother for a predetermined distance as is shown in FIGS. 3A to 3C. Theview-field image 613 of an object (i.e., a broken-line square) isintermittently moved with respect to the imaging-device matrix bydriving an optical system, in the sequence indicated by FIGS. 3A, 3B,3C, and 3D. The optical image of an object need not be divided by prismsor similar means, unlike in the second-type technique.

[0011] The conventional resolution-increasing techniques, describedabove, are disadvantageous in the following respects.

[0012] The first technique can increase the resolution but to a limiteddegree, for two reasons. First, the number of pixels the existingmanufacturing technology can form in the unit area of the device chip islimited. Second, the smaller a pixel, the less sensitive it is. A largerdevice chip may indeed be used to form more pixels on the chip. With theconventional manufacturing method, however, the ratio of defectivepixels to good ones will increase if many pixels are formed on a largechip. Consequently, solid-state imaging devices having a largeimage-receiving surface can hardly be manufactured with a sufficientlyhigh yield.

[0013] In the second resolution-increasing technique, the image dataitems output from the imaging devices (e.g., four devices) are combinedto produce a single image. To render the reproduced image substantiallyidentical to the original image of the object, the images of the objectparts should neither be spaced apart nor overlap one another. The imageswill be spaced apart or overlap unless the pixels arranged along thatedge of one device which abut on the edge of the next device are spacedby exactly the one-pixel distance from the pixels arranged along thatedge of the next device. The imaging devices therefore need to bepositioned with very high precision during the manufacture of the imageprocessing apparatus. It takes much time to position the devices soprecisely, inevitably reducing the manufacture efficiency and,ultimately, raising the cost of the image processing apparatus.

[0014] Also in the resolution-increasing technique similar to thesecond-type technique, the imaging devices must be positioned with highprecision. In addition, the optical system must be driven with highprecision in order to intermittently move the view-field image of anobject (i.e., a broken-line square) with respect to the imaging-devicematrix. A high-precision drive is indispensable to the image processingapparatus. The use of the drive not only makes it difficult tominiaturize or lighten the apparatus, but also raises the manufacturingcost of the apparatus.

[0015] A color image processing apparatus is known, a typtical exampleof which is a so-called “three-section color camera.” This color cameracomprises a color-component generating system and three imaging devices.The color-component generating system decomposes an input optical imageof an object into a red image, a green image, and a blue image. Thethree imaging devices convert the red image, the green image, and theblue image into red signals, green signals, and blue signals—all beingtelevision signals of NTSC system or the like. The signals output fromthe three imaging devices are combined, whereby the red, green and blueimages are combined, forming a single color image of the object. A colorimage with no color distortion cannot be formed unless the imagingdevices are positioned or registered with high precision.

[0016] Images of parts of an object are combined, also in an imageprocessing apparatus which has a plurality of optical imaging devicesfor photographing the parts of the object on photographic film, therebyforming a panoramic image of the object. To form a high-qualitypanoramic image, the images of the object parts should neither be spacedapart nor overlapping one another. Hence, the optical systemincorporated in this image processing apparatus must be controlled withhigh precision. Consequently, the apparatus requires a complex devicefor controlling the optical system, and cannot be manufactured at lowcost.

SUMMARY OF THE INVENTION

[0017] Accordingly it is the object of this invention is to provide animage processing apparatus in which either images of the parts of anobject or images of an object which are identical but different incolor, and for combining the images into a wide high-resolution image ofthe object.

[0018] In a first aspect of the invention, there is provided an imageprocessing apparatus for combining a plurality of images into a singlelarge image such that the images have overlap regions, comprising: imagestoring means for storing image data items representing the images;interpolation means for detecting a positional relation between areference pixel and a given pixel in the overlap area of each image fromimage data read from the image storing means and representing theoverlap area, and for interpolating the image data item read from theimage storing means and representing the image, in accordance with adisplacement coefficient indicating the positional relation, thereby togenerate interpolated image data; and image-synthesizing means forcombining the interpolated image data items generated by theinterpolation means, thereby to form a single large image.

[0019] In a second aspect of the invention, there is provided an imageprocessing apparatus for combining a plurality of images into a singlelarge image such that the images have overlap regions, comprising: lightsplitting means for splitting an object image; a plurality of imagingdevices arranged such that an imaging area of each overlaps that ofanother; image storing means for storing image data items generated bythe imaging devices and representing images overlapping one another andoverlap regions of the images; displacement detecting means fordetecting displacement (i.e., a displacement coefficient consisting of arotation angle R and a parallel displacement S) representing a relationbetween a reference pixel and a given pixel in the overlap area of eachimage from the image data item read from the image storing means andrepresenting the overlap area; interpolation means for interpolating theimage data items read from the image storing means, in accordance withthe rotation angle R and the parallel displacement S detected by thedisplacement detecting means, thereby to generate interpolated imagedata items; and image-synthesizing means for combining the interpolatedimage data items generated by the interpolation means, thereby to form asingle large image.

[0020] In a third aspect of the invention, there is provided an imageprocessing apparatus for combining a plurality of images into a singlelarge image such that the images have overlap regions, comprising:imaging means for intermittently scanning parts of an object image,thereby generating a plurality of image data items; image storing meansfor sequentially storing the image data items generated by the imagingmeans; reference image storing means storing an image data itemrepresenting a reference image; motion vector detecting means forcomparing each image data item read from the image storing means withthe image data item read from the reference image storing means, therebydetecting correlation between the reference image and the imagerepresented by the image data item read from the image storing means anddetecting a motion vector; and image-synthesizing means for processingthe image data items stored in the image storing means, in accordancewith the motion vectors detected by the motion vector detecting means,thereby combining the image data items.

[0021] In a fourth aspect of this invention, there is provided an imageprocessing apparatus for combining a plurality of images into a singlelarge image such that the images have overlap regions, comprising: imagestoring means for storing image data items; a plurality of display meansfor displaying images represented by the image data items read from theimage storing means; interpolation means for interpolating the imagedata items in accordance with displacement coefficients for the displaymeans, thereby generating interpolated image data items representingimages which are to be displayed by the display means, adjoining oneanother without displacement; and image-synthesizing and displayingmeans for combining the image data items stored in the image storingmeans and for displaying the images represented by the image data itemsand adjoining one another without displacement.

[0022] Additional objects and advantages of the invention will be setforth in the description which follows, and in part will be obvious fromthe description, or may be learned by practice of the invention. Theobjects and advantages of the invention may be realized and obtained bymeans of the instrumentalities and combinations particularly pointed outin the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in andconstitute a part of the specification, illustrate presently preferredembodiments of the invention, and together with the general descriptiongiven above and the detailed description of the preferred embodimentsgiven below, serve to explain the principles of the invention.

[0024]FIG. 1 is a diagram showing the positional relation of the opticalsystem and the imaging devices—all incorporated in a conventional imageprocessing apparatus;

[0025]FIG. 2 is a diagram showing the specific positions the imagingdevices assume in a conventional image processing apparatus, in order tocover the predetermined parts of the object;

[0026]FIGS. 3A to 3D are diagrams explaining how a view-field image ofan object is intermittently moved with respect to four imaging devicesin a conventional image processing apparatus;

[0027]FIGS. 4A and 4B are a block diagram and a diagram, respectively,showing the basic structure and operation of an image processingapparatus according to the invention;

[0028]FIG. 5A is a diagram showing the imaging areas of the two CMDsincorporated in the apparatus shown in FIG. 4A, and FIG. 5B is a diagramshowing the positional relation which each screen pixel has with thenearest four CMD pixels;

[0029]FIG. 6 is a block diagram showing an image processing apparatusaccording to a first embodiment of the present invention;

[0030]FIG. 7 is also a block diagram showing the displacement-detectingcircuit and the interpolation circuit, both incorporated in theapparatus shown in FIG. 6;

[0031]FIG. 8 is a diagram illustrating the imaging areas of the CMDsused in the apparatus of FIG. 6, which overlap each other in part;

[0032]FIG. 9 is a diagram representing two displacements vectorsresulting from the rotation and parallel movement of one CMD imagingarea with respect to the other CMD imaging area, respectively;

[0033]FIG. 10 is a diagram explaining how the displacement-detectingcircuit shown in FIG. 7 executes correlation;

[0034]FIG. 11 is a diagram illustrating the positional relation aspecified screen pixel has with four CMD pixels located around thescreen pixel;

[0035]FIG. 12 is a diagram showing the position of each pixel in the CMD8 and that of the corresponding pixel of the CMD 9, in terms of vectors;

[0036]FIG. 13 is a block diagram which shows the displacement-detectingcircuit and the interpolation circuit, both incorporated in an imageprocessing apparatus according to a second embodiment of the presentinvention;

[0037]FIGS. 14A and 14B are diagrams showing the light-splitting sectionof an image processing apparatus according to a third embodiment of thisinvention;

[0038]FIG. 15 is a diagram representing the light distributions in theimaging areas of the CMDs 8 and 9 used in the third embodiment;

[0039]FIG. 16 is a diagram illustrating the light distributions whichhave been obtained by applying an inverse function to different lightdistributions;

[0040]FIG. 17 is a block diagram showing the image processing apparatusaccording to the third embodiment of the invention;

[0041]FIG. 18 is a block diagram showing an image processing apparatusaccording to a fourth embodiment of the present invention;

[0042]FIGS. 19A, 19B, and 19C are diagrams explaining how an input lightflux applied may be applied through separator lenses in various manners,in the apparatus shown in FIG. 18.

[0043]FIG. 20 is a block diagram showing an image processing apparatusaccording to a fifth embodiment of the present invention;

[0044]FIG. 21 is a diagram showing the imaging areas of the two CMDsincorporated in the apparatus shown in FIG. 20;

[0045]FIG. 22A is a block diagram showing an image processing apparatusaccording to a sixth embodiment of this invention;

[0046]FIG. 22B is a diagram explaining how the CMDs are arranged in theapparatus of FIG. 22A;

[0047]FIGS. 23A to 23D are perspective views of four alternativelight-spitting sections for use in an image processing apparatusaccording to a seventh embodiment of the present invention;

[0048]FIGS. 24A and 24B are a side view and a top view, respectively, ofthe light-splitting section shown in FIG. 23A;

[0049]FIGS. 25A and 25B are a side view and a top view, respectively, ofthe light-splitting section shown in FIG. 23B;

[0050]FIG. 26 is a diagram representing the imaging areas of the CMDsused in the seventh embodiment, and also the display area of the displaysection incorporated in the seventh embodiment;

[0051]FIG. 27 is a perspective view showing an image processingapparatus according to an eighth embodiment of the present invention;

[0052]FIGS. 28 and 29 are a plan view and a sectional view,respectively, explaining the first method of positioning CMDs;

[0053]FIG. 30 is a side view of a CMD ceramic package having protrudingmetal terminals;

[0054]FIG. 31 is a side view of a CMD ceramic package comprising asubstrate and spacers mounted on both edges of the substrate;

[0055]FIG. 32 is a plan view, explaining a method of positioning bareCMD chips on a ceramic substrate;

[0056]FIGS. 33A to 33C are views, explaining a method of positioningCMDs, which is employed in the six embodiment of the invention;

[0057]FIG. 34 is a side view, also explaining another method ofpositioning bare CMD chips on a ceramic substrate;

[0058]FIG. 35 is a block diagram showing an image processing apparatusaccording to a ninth embodiment of the present invention;

[0059]FIG. 36 is a block diagram illustrating an image-synthesizingcircuit incorporated in the ninth embodiment;

[0060]FIG. 37 is a diagram explaining the linear interpolation theimage-synthesizing circuit performs;

[0061]FIG. 38 is a block diagram showing an image-synthesizing circuitwhich may be used in the ninth embodiment;

[0062]FIG. 39 is a diagram explaining the linear interpolation which thecircuit shown in FIG. 38 performs;

[0063]FIG. 40 is a block diagram showing an image processing apparatusaccording to a tenth embodiment of the present invention;

[0064]FIG. 41 is a block diagram showing a modification of the apparatusshown in FIG. 40;

[0065]FIGS. 42A, 42B, and 43C are diagrams showing various operatorswhich are used as weighting coefficients in the apparatus shown in FIG.40;

[0066]FIG. 42D is a diagram of edge-emphasizing circuit 127 as shown inFIG. 40;

[0067]FIGS. 43 and 44 are block diagram showing an image processingapparatus according to an eleventh embodiment of this invention;

[0068]FIGS. 45A, 45B, and 45C are diagrams showing three alternativereference patterns which are alternatively used in the eleventhembodiment;

[0069]FIGS. 46A and 46B are diagrams showing two types of referencepattern filters which are alternatively incorporated in an imageprocessing apparatus according to a twelfth embodiment of the invention;

[0070]FIG. 47 is a block diagram showing the apparatus which is thetwelfth embodiment of this invention;

[0071]FIGS. 48A and 48B are diagrams explaining how a synthesized imageis rotated with respect to another image before being combined with theother image;

[0072]FIG. 49 is a block diagram showing an image processing apparatusaccording to a thirteenth embodiment of the invention, in which asynthesized image is rotated as shown in FIGS. 48A and 48B;

[0073]FIGS. 50A and 50B are diagrams explaining how to eliminate anundesirable portion from the adjoining area of a synthesized image, inthe process of combining three or more images into a single image;

[0074]FIG. 51 is a block diagram illustrating an image processingapparatus according to a fourteenth embodiment of the invention, inwhich an undesirable portion is eliminated from the adjoining area of asynthesized image as is shown in FIGS. 50A and 50B;

[0075]FIG. 52 is a block diagram showing a first-type synthesis sectionincorporated in an image processing apparatus according to a fifteenthembodiment of the invention;

[0076]FIG. 53 is a diagram showing the apparatus which is the fifteenthembodiment of the present invention;

[0077]FIG. 54 is a block diagram showing one of identical second-typesynthesis sections used in the apparatus shown in FIG. 53;

[0078]FIG. 55 is a diagram showing an image processing apparatusaccording to a sixteenth embodiment of the present invention;

[0079]FIG. 56 is a block diagram showing one of the identical third-typesynthesis sections used in the sixteenth embodiment;

[0080]FIG. 57 is a side view of a projector which is a seventeenthembodiment of the invention;

[0081]FIG. 58 is a block diagram of the imaging section of the projectorshown in FIG. 57;

[0082]FIG. 59 is a perspective view showing the half prism and thecomponents associated therewith—all incorporated in the projector;

[0083]FIG. 60 is a block diagram showing the system incorporated in theprojector, for detecting the displacements of the LCDs used in theprojector;

[0084]FIG. 61 is a block diagram showing another system which may beused in the projector, to detect the displacement of the LCDs;

[0085]FIG. 62 is a CRT monitor according to the present invention;

[0086]FIG. 63 is a block diagram of a film-editing apparatus which is aneighteenth embodiment of this invention;

[0087]FIGS. 64A to 64E are diagrams various positions the line sensorsmay assume in the apparatus shown in FIG. 63, and showing the conditionof an image formed;

[0088]FIGS. 65A and 65B are block diagrams showing, in detail, an imageprocessing apparatus according to a nineteenth embodiment of theinvention;

[0089]FIG. 66 is a block diagram illustrating an image processingapparatus which is a twentieth embodiment of the present invention;

[0090]FIG. 67 is a block diagram showing an electronic camera which is atwenty-first embodiment of this invention;

[0091]FIG. 68 is a block diagram showing the shake-correcting circuitincorporated in the electronic camera of FIG. 67;

[0092]FIGS. 69A to 69D are diagrams explaining how the imaging area ofthe camera (FIG. 67) moves, without shaking, with respect to the imageof an object;

[0093]FIGS. 70A to 70D are diagrams illustrating how the imaging area ofthe camera moves, while shaking, with respect to the image of an object;

[0094]FIG. 71 is a diagram explaining the method of finding thecorrelation between a reference image and an object image by moving theobject image with respect to the reference image;

[0095]FIGS. 72A and 72B are diagrams explaining how to determine thedistance and angle by which an image has moved and rotated;

[0096]FIG. 73 is a diagram showing how an image is moved;

[0097]FIGS. 74A and 74B are perspective views of the electronic camera(FIG. 67) and a recording section, explaining how to operate the camerain order to form an image of an object and record the image;

[0098]FIG. 75 is a diagram showing the imaging section of an electroniccamera which is a twenty-second embodiment of the invention;

[0099]FIGS. 76A and 76B are diagrams explaining the technique which isemployed in a twenty-third embodiment of the invention in order tocalculate the correlation between images with high accuracy;

[0100]FIG. 77 is a block diagram showing a shake-correcting circuit foruse in a twenty-fourth embodiment of the invention;

[0101]FIG. 78 is a block diagram showing the correlated area selectorincorporated in the circuit illustrated in FIG. 77;

[0102]FIG. 79 is a diagram showing images one of which may be selectedby the image-selecting circuit incorporated in the correlated areaselector shown in FIG. 78;

[0103]FIGS. 80A, 80B, and 80C show three sets of coefficients for aconvolution filter;

[0104]FIG. 81 is a circuit for obtaining the absolute sum of the valuedifferences among adjacent pixels;

[0105]FIG. 82 is a side view showing the imaging section of anelectronic camera;

[0106]FIG. 83 is a side view illustrating another type of an imagingsection for use in the electronic camera;

[0107]FIG. 84 is a cross-sectional side view of the imaging section ofan electronic camera which is a twenty-fifth embodiment of theinvention;

[0108]FIG. 85 is a circuit diagram showing the CMD incorporated in theimaging section of FIG. 84;

[0109]FIG. 86 is a block diagram of the processing section used in theimaging section shown in FIG. 84;

[0110]FIGS. 87A and 87B are a timing chart representing the timing oflight-emission at the stroboscopic lamp incorporated in the electroniccamera shown in FIG. 84;

[0111]FIG. 88 is a cross-sectional side view of the imaging section ofan electronic camera which is a twenty-sixth embodiment of theinvention;

[0112]FIG. 89 is a timing chart explaining how the mirror isintermittently driven in the imaging section shown in FIG. 88;

[0113]FIGS. 90A and 90B are cross-sectional side views of the imagingsection of an electronic camera which is a twenty-seventh embodiment ofthe invention, and FIG. 90C is a chart representing the timing ofexposure performed in the imaging section of FIGS. 90A and 90B;

[0114]FIG. 90D is a cross-sectional view of the cam of FIG. 90A;

[0115]FIG. 90E is a plan view of the screw of FIG. 90B;

[0116]FIG. 91 is a block diagram illustrating the imaging section of anelectronic camera which is a twenty-eighth embodiment of the invention;

[0117]FIG. 92 is a block diagram showing an ultrasonic diagnosisapparatus which is a twenty-ninth embodiment of this invention and whichis a modification of the embodiment shown in FIG. 91;

[0118]FIG. 93 is a block diagram showing the imaging section of thetwenty-ninth embodiment;

[0119]FIG. 94 is a diagram showing a convex-type ultra-sonic image;

[0120]FIGS. 95A and 95B are diagrams explaining how to combine twoimages in the twenty-ninth embodiment of the invention;

[0121]FIG. 96 is a diagram how to synthesize an image;

[0122]FIGS. 97A, 97B, and 97C are diagrams illustrating the imagingsection of an electronic camera which is a thirtieth embodiment of thepresent invention;

[0123]FIG. 98 is a block diagram showing an apparatus for reproducingthe image taken by the imaging section shown in FIGS. 97A, 97B, and 97C;

[0124]FIG. 99 is a block diagram showing, in detail, theimage-synthesizing circuit incorporated in the apparatus of FIG. 98;

[0125]FIG. 100 is a diagram explaining how three images overlap and howthe coefficients for the overlap regions change;

[0126]FIG. 101 is a block diagram illustrating the image-adding sectionincorporated in the imaging section of FIG. 97C;

[0127]FIG. 102 is a block diagram showing an electronic camera which isa thirty-first embodiment of the invention;

[0128]FIG. 103 is a diagram showing the field of the view finder view ofthe camera illustrated in FIG. 102;

[0129]FIG. 104A is a diagram explaining how to combine a plurality ofimages into a wide image in a thirty-second embodiment of the invention;

[0130]FIG. 104B is a diagram showing the field of the view finder of thecamera used in the thirty-second embodiment;

[0131]FIGS. 105A and 105B are side views showing an electronic camerawhich is a thirty-third embodiment of the invention and which is used toread data from a flat original;

[0132]FIG. 106 is a block diagram showing an image processing apparatusaccording to a thirty-fourth embodiment of the present invention;

[0133]FIG. 107 is a plan view showing the photosensitive film used inthe apparatus of FIG. 106;

[0134]FIG. 108 is a diagram illustrating an address signal recorded onthe magnetic tracks of the film shown in FIG. 107;

[0135]FIGS. 109A, 109B, and 109C are diagrams showing the positionswhich recorded images assume on the imaging area of the film;

[0136]FIG. 110 is a block diagram showing an image processing apparatusaccording to a thirty-fifth embodiment of the present invention;

[0137]FIG. 111 is a perspective view showing the imaging section of theapparatus shown in FIG. 110;

[0138]FIG. 112 is a block diagram showing an image processing apparatusaccording to a thirty-sixth embodiment of the present invention;

[0139]FIG. 113 is a diagram illustrating an address signal recorded onthe magnetic tracks of the film used in the apparatus of FIG. 112;

[0140]FIGS. 114A and 114B is a block diagram showing an image processingapparatus according to a thirty-seventh embodiment of the presentinvention;

[0141]FIG. 115 is a diagram showing the interpolation circuitincorporated in the apparatus of FIG. 114;

[0142]FIG. 116A is a diagram showing the reference areas used fordetecting the displacement of a G image;

[0143]FIG. 116B is a diagram showing areas which are searched for thatpart of a R or B image which corresponds to a predetermined part of theG image;

[0144]FIG. 117 is a diagram illustrating displacement vectors detectedand processed in the thirty-seventh embodiment;

[0145]FIG. 118 is a diagram showing, in detail, one of the identicalcorrelation circuits used in the apparatus of FIG. 114;

[0146]FIG. 119 is a diagram explaining how a pixel value is interpolatedin the apparatus of FIG. 114;

[0147]FIG. 120 is a diagram showing, in detail, one of the identicalcoefficient calculators incorporated in the apparatus of FIG. 114;

[0148]FIG. 121 is a diagram showing, in detail, one of the identicalcoefficient memories of the apparatus shown in FIG. 114;

[0149]FIG. 122 is a diagram showing an imaging area in which a R image,a G image, and a B image overlap one another;

[0150]FIGS. 123A and 123B is a block diagram illustrating an imageprocessing apparatus according to a thirty-eighth embodiment of theinvention;

[0151]FIG. 124 is a diagram showing a coefficient calculatorincorporated in an image processing apparatus according to athirty-ninth embodiment of the present invention;

[0152]FIG. 125 is a diagram showing a coefficient memory used in thethirty-ninth embodiment; and

[0153]FIG. 126 is a diagram illustrating one of L x L blocks of animage.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0154] The basic structure and operation of an image processingapparatus according to the present invention will be described, withreference to FIGS. 4A and 4B and FIGS. 5A and 5B.

[0155] As FIG. 4A shows, the apparatus comprises a light-splittingsection, an imaging section 2, an image-storing section 3, adisplacement-detecting section 4, an interpolation section 5, anobjective lens 6, a image-synthesizing section 7, and a display section31. The imaging section 2 has two CMDs 8 and 9 (i.e., solid-stateimaging devices). As shown in FIG. 4B, the CMDs 8 and 9 are positionedwith such precision that they receive two parts of the optical imagewhich overlap in part.

[0156] In operation, the objective lens 6 applies an optical image of anobject (not shown) to the light-splitting section 1. The section 1splits the input light into two parts representing two parts of theimage which overlap in part. The parts of the image are applied to theCMDs 8 and 9 of the imaging section 2. The CMDs 8 and 9 convert theimage parts into image signals, which are supplied to the image-storingsection 3. The section 3, e.g., a frame memory, temporarily stores theimage signals.

[0157] Then, the displacement-detecting section 4 detects the positionalrelation between one of the pixels of either CMD (Charge ModulationDevices) and the corresponding pixel of the screen of the displaysection 31, from the image signal read from the image-storing section 3and representing the overlap regions d of the two image parts shown inFIG. 5A, wherein the black dots indicate the pixels of the CMDs and thewhite dots indicate the pixels of the display screen. More specifically,the section 4 performs correlation on the value of each CMD pixel,thereby calculating two conversion factors, i.e., rotation angle R anddisplacement S.

[0158] In accordance with the conversion factors, the interpolationsection 5 interpolates the value of each screen pixel from the values ofthe CMD pixels located near the screen pixel, thereby producing aninterpolated pixel signal representing the screen pixel. Thus, theinterpolation section 5 outputs the interpolated pixel signalsrepresenting all pixels of the display section 31, to theimage-synthesizing section 7.

[0159] The image-synthesizing section 7 combines the image signalsproduced by the interpolation section 5 with the image signals read fromthe image-storing section 3, thereby generating image signals whichrepresent a single continuous image of the object. These image signalsare supplied to the display section 31. The section 31 displays ahigh-resolution image of the object.

[0160] Schematically shown in FIG. 5A are the imaging area a (i.e., anM×N pixel matrix) of the CMD 8, the imaging area b (i.e., an M×N pixelmatrix) of the CMD 9, and the display area c of the display section 31.As is evident from FIG. 5A, the display area c is an (u+v)×w pixelmatrix and is completely covered with the imaging areas a and b whichoverlap at regions d. In the instance shown in FIG. 5A, each pixel ofthe display area c assumes the same position as the corresponding pixelof the imaging area a.

[0161] As has been described, the displacement-detecting section 4detects the positional relation (i.e., rotation angle R and displacementS), between each pixel of the imaging area b and the corresponding pixelof the display area c, from the image signals read from theimage-storing section 3 and representing the overlap regions d. Todetect the positional relation, the section 4 needs the values of thepixels d11, d21, . . . d (u+v)w of the display area c—all indicated bywhite dots. For the values of the pixels dij (i=1 to u, j=1 to w) of thedisplay area c, the values of the pixels of CMD 8 are utilized. Thevalue for each of the remaining pixels A of the display area c, i.e.,the pixels dij (i=u+1 to u+v, j=1 to w), is interpolated from the valuesof the four pixels B, C, D and E of the imaging area b which surroundthe pixel dij, as is illustrated in FIG. 5B.

[0162] In order to calculate the value for any desired pixel of thedisplay area c, it suffices to position the CMDs 8 and 9 with suchprecision that their imaging areas a and b completely cover the displayarea c of the display section 31 and overlap in part appropriately. Evenif the pixels of either imaging area are deviated from the correspondingpixels of the display area c for a distance of several pixels, theapparatus can form a high-resolution single image of an object. It istherefore not necessary to position the CMDs 8 and 9 with high precisionon the order of a one-pixel distance as in the conventional imageprocessing apparatus. Hence, the image processing apparatus according tothe invention can be easily manufactured, and its manufacturing cost canbe low.

[0163] An image processing apparatus, which is a first embodiment of theinvention, will now be described, with reference to FIGS. 6 to 12.

[0164] The apparatus has a half prism 1 a comprised of two right-angleprisms connected together. Two CMDs 8 and 9 (i.e., two-dimensionalsolid-state imaging devices) 8 and 9 are mounted on the top and back ofthe half prism la, respectively. The CMDs 8 and 9 are positioned suchthat their imaging areas overlap in part.

[0165] To the half prism 1 a, an optical system 6 applies light whichrepresents an image of an object (not shown). The half prism 1 a splitsthe input light into two parts. The parts of the input light are appliedto the CMDs 8 and 9. Each of the CMDs 8 and 9 converts the input lightinto an image signal, under the control of a CMD driver 32.

[0166] The image signals output by the CMDs 8 and 9 are supplied topre-amplifiers 10 and 11, which amplify the signals. Low-pass filters(LPFs) 12 and 13 remove noise components from the amplified imagesignals. The signals output by the filters 12 and 13 are input to A/Dconverters 14 and 15, respectively. The A/D converters 14 and 15 convertthe input signals into digital image signals, which are supplied tosubtracters 16 and 17.

[0167] The FPNs (Fixed Pattern Noises) of the CMDs 8 and 9, stored inFPN memories 18 and 19, are supplied to the subtracters 16 and 17,respectively. The subtracter 16 takes the FPN of the CMD 8 from theimage signal output from the A/D converter 14. Similarly, the subtracter17 takes the FPN of the CMD 9 from the image signal output from the A/Dconverter 15. The image signals output by the subtracters 16 and 17 areinput to signal processors (SPs) 20 and 21, which perform y correctionor outline emphasis on the input image signals.

[0168] The image signals processed by the processors 20 and 21 arestored into frame memories 22 and 23, respectively. At a proper time,the image signals are read from the frame memories 22 and 23 andsupplied to a displacement-detecting circuit 24. The circuit 24 detectsthe displacement of the overlap regions of the imaging areas of the CMDs8 and 9. The displacement is defined by two conversion factors R and S.The factors R represents the rotation matrix R which one CMD imagingarea has with respect to the other CMD imaging area. The factors Srepresents the displacement vector which results from a parallelmovement of one CMD imaging area with respect to the other CMD imagingarea.

[0169] The displacement, or the conversion factor R and S, are suppliedfrom the circuit 24 to an interpolation circuit 25. The circuit 25interpolates the pixel values read from the frame memory 23 inaccordance with the conversion factors R and S. The pixel values, thusinterpolated, are input to a parallel-serial (PS) converter 29, alongwith the signals read from the frame memory 22. The converter 29converts the pixel values and the signals into serial signals. Theserial signals are written into a frame memory 30 and read therefrom toa display section 31. The display section 31 displays a high-resolutionsingle image of the object.

[0170] The image processing apparatus has a system controller 33. Thecontroller 33 controls the FPN memories 18 and 19, the frame memories 22and 23, the interpolation circuit 25, the PS converter 29, and the CMDdriver 32.

[0171] The displacement-detecting circuit 24 and the interpolationcircuit 25 will be described in detail, with reference to FIG. 7.

[0172] The displacement-detecting circuit 24 comprises correlators 24 aand 24 b and a coefficient calculator 24 c. The correlators 24 a and 24b receive the image signals read from the frame memories 22 and 23,respectively, and perform correlation on the input image signals. Theimage signals, thus processed, are input to the coefficient calculator24 c. The calculator 24 c detects the displacement of the overlapregions of the CMD imaging areas, i.e., the conversion factors R and S.

[0173] The conversion factors R and S are stored into the memories 26and 27 incorporated in the interpolation circuit 25. In theinterpolation circuit 25, the factors R and S read from the memories 26and 27 are input to a coordinates-converting circuit 35.

[0174] The coordinates value X1 of the point designated by the systemcontroller 33 is input via a coordinates selector 34 to thecoordinates-converting circuit 35. The circuit 35 converts thecoordinates value X₁ to a coordinate value X₂, using the conversionfactors R and S, in accordance with a predetermined conversion formula(10) which will be described later. The coordinate value X₂ pertains tothe imaging area of the CMD 9. The value X₂ is supplied from thecoordinates-converting circuit 35 to a data-reading circuit 36 and aninterpolation coefficient calculator 37.

[0175] From the coordinate value X₂ the data-reading circuit 36 producespixel values v_(b), v_(c), v_(d), and v_(e), which are input to a linearinterpolation circuit 38. Meanwhile, the interpolation coefficientcalculator 37 calculates interpolation coefficients a, b, c, and d fromthe coordinate value X₂ and inputs these coefficients a, b, c, and d tothe linear interpolation circuit 38. In the linear interpolation circuit38, the pixel values v_(b), v_(c), v_(d), and v_(e) are supplied to fourmultipliers 39, respectively, and the interpolation coefficients a, b,c, and d are supplied also to the multipliers 39, respectively. Thefirst multiplier 39 multiples the pixel value v_(b) by the coefficienta; the second multiplier 39 multiples the pixel value v_(c) by thecoefficient b; the third multiplier 39 multiples the pixel value v_(d)by the coefficient c; and the fourth multiplier 39 multiples the pixelvalue v_(e) by the coefficient d. Further, in the linear interpolationcircuit 38, the outputs of the multipliers 39 are input to an adder 40which adds the outputs of the multipliers 39, generating aninterpolation value v_(a).

[0176] To obtain the conversion factors R and S it is required that areference point be set for the rotation and parallel movement of one ofthe CMD imaging areas with respect to the other CMD imaging area. In thefirst embodiment, as FIG. 8 shows, the reference point is the center C₁of an overlap area 1, i.e., those portions of the imaging areas of theCMDs 8 and 9 which will overlap one another if the CMDs 8 and 9 arepositioned precisely. In practice, the CMDs 8 and 9 cannot be positionedprecisely, and an overlap area 2, i.e., the mutually overlappingportions of the imaging areas of the CMDs 8 and 9, has a center C₂ whichis displaced from the center C₁ for a distance corresponding to theconversion factor S. As can be understood from FIG. 8, the overlap area2 is rotated around the center C₂ with respect to the overlap area 1 byan angle corresponding to the conversion factor R.

[0177] The conversion factors S and R can be obtained from, for example,displacement vectors v₁ and v₂ in the overlap area 1 which are atpositions P₁ and P₂ which S are symmetrical with respect to the centerC₁. The vectors v₁ and v₂ are presented by the following equations (1)and (2), respectively, because of the vectors r and s which result fromthe rotation of the imaging area the CMD 9 with respect to that of theCMD 8:

Vector v ₁=vector r+vector s

Vector v ₂=−(vector r)+vector s   (1)

[0178] where the vectors s and r are given as:

|vector r|=L tan θ  (2)

[0179] Therefore, the vectors s and r are:

Vector s=(v ₁ +v ₂)/2   (3)

Vector r=(v ₁ −v ₂)/2   (4)

[0180] The rotation matrix R of the imaging area of the CMD 9 withrespect to that of the CMD8 is represented by the following equation:$\begin{matrix}{R = \begin{pmatrix}{{\cos \quad \theta} - {\sin \quad \theta}} \\{{\sin \quad \theta} - {\cos \quad \theta}}\end{pmatrix}} & (5)\end{matrix}$

[0181] Angle θ is found from the equation (2); as follows:

θ=tan⁻¹(vector r)/L   (6)

[0182] In the equation (6), L is a known amount, and vector r isdetermined by the equation (4). Hence, angle θ can be found, and therotation matrix R can also be obtained. The rotation matrix R and thedisplacement vector S (i.e., the vector of the parallel displacement ofthe imaging area of the CMD 9), thus calculated from the displacementvectors v₁ and v₂ at positions P₁ and P₂, are stored as conversionfactors R and S in the memories 26 and 27, respectively.

[0183] The correlation the correlators 24 a and 24 b execute on theinput image signals may be one of the various types hitherto known. Inthis embodiment, the correlation is effected as is shown in FIG. 10.That is, the correlator 24 a detects the position (x₁, y₁) where theabsolute sum of the reference area r₁ and r₂ of the CMD 8 is minimum inthe search area S₁ of the CMD 9, and the correlator 24 b detects theposition (x₂, Y₂) where the absolute sum of the reference area r₁ and r₂of the CMD 8 is minimum in the search area S₂ of the CMD 9. Thecoordinates values of these positions, (x₁, y₁) and (x₂, Y₂), are inputto the coefficient calculator 24 c. The calculator 24 c performs theoperations of the equations (3), (4), (6), and (5) sequentially on thecoordinates values (x₁, y₁) and (x₂, Y₂), obtaining the rotation matrixR and the displacement vector S. The displacement vector S=(S_(x),S_(y)).

[0184] The operation of the interpolation circuit 25 will be explained,with reference to FIGS. 6 and 7.

[0185] The interpolation circuit 25 performs linear interpolation on thefour pixel values read from the frame memory 23, thereby finding thevalue of the pixel at the position designated by the system controller33, as will be described with reference to FIG. 11. First, the valuev_(a) of the pixel A is obtained from the values v_(b), v_(c), v_(d),and v_(e) of the four pixels B, C, D, and E which are located around thepixel A. More precisely, the values of v_(f) and v_(g) which the pixelslocated at the intersections F and G of the vertical line passing thepixel A, the line BC connecting the pixels B and C, and the line DEconnecting the pixels D and E are:

v _(f) =nvb+mb _(c) /m+n   (7-a)

v _(g) =nv _(d) +mv _(e) /m+n   (7-b)

[0186] where BF=DG=m, and FC=GE=n.

[0187] Assuming FA=p and AG=q, then the value va for pixel A can begiven as:

v _(a) =qv _(f) +pv _(g) /p+q   (8)

[0188] If it is assumed that the inter-pixel distance is “1,” then m+n=p+q=1. Hence, from the equations (7-a), (7-b), and (8), the value v_(a)for the pixel A is calculated as follows:

v _(a) =av _(b) +bv _(c) +cv _(d) +dv _(e)   (9)

[0189] where a=(1−p)(1−m), b=(1−p)m, c=p(1−m), and d=pm. Namely, thepixel value va can be obtained directly from m, p, and the values v_(b),v_(c), v_(d), and v_(e) of the four pixels located around the pixel A.

[0190] It will now be explained how to find values for m and p, withreference to FIG. 12. In the first embodiment, m and p are of suchvalues that the center C₁ of the overlap area 1 of the CMD 8 isconsidered the origin of the coordinate, the position any pixel assumesin the overlap area 1 is represented by vector x₁, the center C₂ of theoverlap area 2 of the CMD 9 is regarded as the origin of the coordinate,and the position any pixel assumes in the overlap area 1 is representedby vector x₂. To find m and p, it is necessary to convert thecoordinates of the position represented by the vector x₁ into vector x₂.In other words, coordinate conversion must be carried out. Assuming thatx₁=(i₁, j₁) and x₂=(i₂, j₂), and that the vectors x₁ and x₂ havedifferent coordinate axes, the vector x₂ will then be given as follows:

x ₂=(x ₁ −S)   (10-a)

[0191] where R−1 means the rotation by angle of −θ. In terms of thecomponents of the vectors x nd x, the equation (10-a) changes to:$\begin{matrix}{\begin{matrix}i_{2} \\j_{2}\end{matrix} = {\begin{pmatrix}{\cos \quad \theta \quad \sin \quad \theta} \\{{- \sin}\quad \theta \quad \cos \quad \theta}\end{pmatrix}\begin{pmatrix}{i_{1} - S_{x}} \\{j_{1} - S_{y}}\end{pmatrix}}} & \left( {10\text{-}b} \right)\end{matrix}$

[0192] The equation (10-b) shows that the coordinates (i1, j1) in theimaging area of the CMD 8 are equivalent to the following coordinates inthe imaging area of the CMD 9:

(i ₂ , j ₂)={(i ₁ −s _(x))cos θ+(j ₁ −s _(y))sin θ, −(i ₁ −s _(x))sinθ+(j ₁ −s _(y))cos θ}  (10-b)

[0193] The notation of (i₂, j₂) represents real numbers which define thecoordinates of the pixel A shown in FIG. 11. Hence, m and p are givenas:

m=i ₂−(int)i ₂   (11-a)

p=j ₂−(int)j ₂   (11-b)

[0194] where the notation of (int) means integration of numbers.Similarly, the coordinates of the pixels B, C, D, and E are representedas follows:

B=((int)i ₂, (int)j ₂)

C=((int)i ₂+1, (int)j ₂)

D=((int)i₂, (int)j ₂+1)

E=((int)i ₂+1, (int)j ₂+1)   (12)

[0195] The conversion factors R and S calculated as described above arewritten into the memories 26 and 27 of the interpolation circuit 25during the manufacture of the image processing apparatus. Thereafter itis unnecessary for the displacement-detecting circuit 24 to detect theconversion factor R or the conversion factor S. It suffices to read thefactors R and S from the memories 26 and 27, respectively, whenever itis required to do so.

[0196] Therefore, once the conversion factors R and S have been thusstored into the memories 26 and 27 of the interpolation circuit 25, thedisplacement-detecting circuit 24 is no longer necessary in the imageprocessing apparatus. Stated in another was, a user of the apparatusneed not make use of the circuit 24. Usually, the circuit 24 is removedfrom the apparatus and used again in the factory to detect conversionfactors R and S for another apparatus of the same type.

[0197] It will now be explained how to use the image processingapparatus according to the first embodiment of the invention.

[0198] First, the user holds the apparatus at a proper position, thusplacing the image of an object within the view field, which he or shewishes to photograph at high resolution. The user then pulses theshutter-release 10 button of the apparatus, whereby two image signalsare stored into the frame memories 22 and 23. These image signalsrepresent those parts of the optical image applied to the imaging areasof the CMD 8 and 9, respectively.

[0199] Next, the image signals are read from the frame memories 22 and23, ultimately inputting to the frame memory 30 the pixel signalsrepresenting the (u+v)×w pixels, i.e., the pixel d 11 to d (u+v)warranged in the display area c of the display section 31. As is evidentfrom FIG. 5A, the values of the pixels of CMD 8 are utilized for thoseof the pixels d 11 to d_(ij) (i=1 to u, j=1 to w) of the display area c.The value for each of the remaining pixels of the display area c, i.e.,the pixels d_(ij) (i=u+1 to u+v, j=1 to w), is interpolated from thevalues of the four pixels B, C, D and E of the imaging area b of the CMD9 which are located around the pixel of the display area c. Moreprecisely, the system controller 33 designates the coordinates value X₁of any desired pixel d_(ij), and this value X₁ is input to theinterpolation circuit 25. In the circuit 25, the coordinates selector 34selects a coordinates value x₁ representing the position the pixeld_(ij) assumes in the overlap area 1 of the CMD 8. The value x₁, thusselected, is input to the coordinates-converting circuit 35. The circuit35 calculates the coordinates value x₂ of the from the coordinates valuex₁ pertaining to the imaging area a of the CMD 8, using the conversionfactors R and S in accordance with the equation (10). The coordinatevalue x2 is input to both the data-reading circuit 36 and theinterpolation coefficient calculator 37.

[0200] The data-reading circuit 36 calculates, from the coordinate valuex₂, the coordinates of the four pixels B, C, D, and E around the pixel Ain accordance with the equation (12). Then, the circuit 36 reads thepixel values v_(b), v_(c), v_(d), and v_(e) from the frame memory 23,which correspond to the coordinates values thus calculated, and inputsthese pixel values to the linear interpolation circuit 38.

[0201] The interpolation coefficient calculator 37 calculates m and pfrom the coordinate value x₂ in accordance with the equation (11),thereby obtaining interpolation coefficients a, b, c, and d. Thesecoefficients a, b, c, and d are input to the linear interpolationcircuit 38.

[0202] The linear interpolation circuit 38 interpolates the value v_(a)of the pixel d_(ij) from the pixel values v_(b), v_(c), v_(d) and v_(e)and the interpolation coefficients a, b, c and d, in accordance with theequation (9). The coordinates value v_(a), thus calculated, is suppliedto the PS converter 29. The coordinate values for all other desiredpixel d_(ij) are calculated in the same way and input to the PSconverter 29. The PS converter 29 converts the pixel values, which areparallel data, to serial data, or a continuous image signal. Thecontinuous image signal is written at predetermined addresses of theframe memory 30. The image signal is read from the frame memory 30 andsupplied to the display section 31. The display section 31 displays ahigh-resolution single image of the object.

[0203] The value for each pixel d_(ij) may be output to the displaysection 31 from the PS converter 29 immediately after it has beeninterpolated by the interpolation circuit 25. If this is the case, theframe memory 30 can be dispensed with.

[0204] As has been described, the image processing apparatus accordingto the first embodiment of the invention can form a singlehigh-resolution image of na object even if the CMDs 8 and 9 are notpositioned precisely, since the interpolation circuit 25 interpolatesthe value for any desired pixel. Thus, the CMDs 8 and 9 need not bepositioned with high precision, whereby the image processing apparatuscan be manufactured at low cost. Moreover, since the apparatus has nomechanical, movable components, it can be made small and light.

[0205] In the first embodiment, the displacement-detecting circuit 24 isincorporated during the manufacture of the apparatus and is removedtherefrom after the conversion factors R and S are stored into thememories 26 and 27 of the interpolation circuit 25. Instead, the imagingsection of the apparatus can have a connector so that the circuit 24 maybe connected to the imaging section or disconnected therefrom. Further,the interpolation circuit 25 is not limited to the type which executeslinear interpolation. Rather, the circuit 25 may be one which effects ahigher interpolation such as spline interpolation or a sincinterpolation.

[0206] An image processing apparatus, which is a second embodiment ofthe invention will be described, with reference to FIG. 13.

[0207] The first embodiment described above must process a considerablylarge amount of data whenever an image of an object is photographed,performing the calculations based on the equations (9), (10), (11), and(12). The second embodiment is designed not to effect these calculationson the image signals representing each image taken. To be more specific,as shown in FIG. 13, the second embodiment has aninterpolation-coefficient writing circuit 28 and an interpolationcircuit 25 a which replace the displacement-detecting circuit 24 and theinterpolation circuit 25, respectively. The second embodiment isidentical to the first in all other respects. Its components identicalto those of the first embodiment are therefore designated at the samereference numerals in FIG. 13 and will not described in detail in thefollowing description.

[0208] As shown in FIG. 13, the interpolation-coefficient writingcircuit 28 comprises a displacement-detecting circuit 24, acoordinates-converting circuit 35, an interpolation coefficientcalculator 37, and a data-address detector 41. Thecoordinates-converting circuit 35 performs the operation of the equation(10), the interpolation coefficient calculator 37 effects the operationsof the equations (11) and (9), and the data-address detector 41 executesthe operation of the equation (12). The circuit 37 calculatesinterpolation coefficients a, b, c, and d. The detector 41 detects thecoordinates value of each pixel. The coefficients a, b, c, and d, andthe coordinate value of the pixel are input to the interpolation circuit25 a.

[0209] In the interpolation circuit 25 a, the coordinate value of thepixel is stored into a data address memory 42, and the interpolationcoefficients a, b, c, and d are stored into four coefficient memories43, 44, 45, and 46, respectively. The circuit 25 a further comprises acoordinates selector 34, a data-reading circuit 36 b, and a linearinterpolation circuit 38.

[0210] As indicated above, the second embodiment effects the coordinateconversion of the equation (10), the interpolation-coefficientcalculation of the equation (11), and the coordinates-value calculationof the equation (12) during the manufacture of the apparatus, and theresults of these operations are stored into the data-address memory 42and the coefficient memories 43 to 46. Hence, it is only the operationof the equation (9) that the linear interpolation circuit 38 needs toaccomplish.

[0211] The use of the data-address memory 42 and the coefficientmemories 43 to 46, all incorporated in the interpolation circuit 25 a,greatly reduce the amount of data that needs to be processed. Thisenables the apparatus to process, at sufficiently high speed, the imagesignals which are sequentially generated by continuous imaging.

[0212] In the second embodiment, the interpolation-coefficient writingcircuit 28 may be connected to the apparatus only during the manufactureof the apparatus, and may be disconnected therefrom after the operationsof the equations (10), (11) and (12) are performed and the resultsthereof are stored into the data-address memory 42 and the coefficientmemories 43 to 46.

[0213] An image processing apparatus, which is a third embodiment ofthis invention, will be described with reference to FIGS. 14A and 14Band FIGS. 15 to 17. The third embodiment is similar to the firstembodiment shown in FIG. 6, and the same components as those of the 5first embodiment are denoted at the same reference numerals in FIG. 17and will not be described in detail.

[0214] In the first and second embodiments, the input light representingthe optical image of an object is split into two parts by means of thehalf prism 1 a. The use of the half prism 1 a is disadvantageous in thatone half of the input light is wasted. In the third embodiment, to avoidwasting of the input light, one of the prisms constituting thelight-splitting section has a coating on a part of its output surface asis shown in FIG. 14A. Thus, the portions of the first prism havedifferent transmission coefficients.

[0215]FIG. 14A shows an input light flux which is coaxial with theoptical axis of the light-splitting section. The upper half (shadedpart) of the flux is reflected to a CMD 8 from the coated part of theoutput surface of the first prism, whereas the lower half of the fluxpasses through the first prism and the second prism, reaching a CMD 9.On the other hand, FIG. 14B shows an input light flux whose axisdeviates upwards from the optical axis of the light-splitting section. Agreater upper (shaded) part of the flux is reflected to the CMD 8 fromthe coated part of the output surface of the first prism, whereas thesmaller lower half of the flux passes through the first prism and thesecond prism, forming a small part of the input image on the upper edgeportion of the CMD 9.

[0216] The amount of light input to the light-splitting section isproportional to the area of the output aperture of the objective lens.Thus, when the input light flux is coaxial with the optical axis of thelight-splitting section as is shown in FIG. 14A, the light distributionsin the imaging areas of the CMDs 8 and 9 are symmetrical with respect tothe optical axis of the light-splitting section, as is illustrated inFIG. 15. As is evident from FIG. 15, the light amount at the opticalaxis of the light-splitting section is equal to the amount applied tothe CMDs through the half prism 1 a in the first and second embodiments.The light distributions in the imaging areas of the CMDs 8 and 9 aredifferent, particularly in the overlap areas thereof.

[0217] From the light distributions in the imaging areas of the CMDs 8and 9 which are different, a displacement, if any, of the imaging areaof one CMD with respect to that of the other CMD cannot be detectedcorrectly. Namely, the displacement detected is erroneous. Further, ifthe light distributions on the CMDs differ, the image formed by theapparatus will have brightness distortion. In order to prevent suchbrightness distortion, some measures must be taken to render the lightdistributions on the CMDs equal.

[0218] In the third embodiment, use is made of light-amount correctingcircuits 47 and 48 as shown in FIG. 17. These circuits 47 and 48 amplifyinput image signals originated from the CMDs 8 and 9, making the lightdistributions on the CMDs equal to each other as shown in FIG. 16. Inother words, the circuits 47 and 48 apply an inverse function to thedifferent light distributions on the CMDs 8 and 9. The light-amountcorrecting circuits 47 and 48 may be look-up tables. The light-splittingsection 1 b of the third embodiment comprises two prisms. One of theprisms has a coating on a part of its output surface as is shown in FIG.14A and consists of two portions having different transmissioncoefficients.

[0219] Consisting of two portions with different transmissioncoefficients, this prism reduces the loss of input light to a minimum,whereby the apparatus is made suitable for photographing dark objects.In the third embodiment, the prisms have each two parts having greatlydifferent transmission coefficients. Each of them may be replaced by aprism which has such a coating that its transmission coefficientgradually changes in one direction.

[0220] An image processing apparatus, which is a fourth embodiment ofthe invention, will be described with reference to FIG. 18 and FIGS.19A, 19B, and 19C. The fourth embodiment is also similar to the firstembodiment (FIG. 6). The components identical to those of the firstembodiment are denoted at the same reference numerals in FIG. 18 andwill not be described in detail.

[0221]FIGS. 19A, 19B, and 19C explain how a light flux applied from anobjective lens 6 is applied through the separator lenses 1 c, forming animage on CMDs 8 and 9 in various manners. To be more specific, FIG. 19Ashows a light flux applied through the lenses 1 c to the CMDs 8 and 9,exactly along the optical axis of the lens 6. FIG. 19B shows a lightflux extending along a line inclined to the optical axis of theobjective lens 6, forming an image on the upper edge portion of the CMD8 only. FIG. 19C shows a light flux extending along a line parallel toand deviating downward from the optical axis of the lens 6, forming animage on the upper edge portion of the CMD 9 only. When the input lightflux is applied as shown in FIG. 19A or 19C, the lenses 1 c split theflux into two parts, and these parts of the flux form images on bothCMDs 8 and 9 or on the CMD 9 only, which overlap in part. A light shield50 is arranged between the separator lenses 1 c and the CMDs 8 and 9,extending in a horizontal plane containing the optical axis of the lens6. Hence, the shield 50 prevents mixing of the two flux parts.

[0222] As can be understood from FIGS. 19B and 19C, the lightdistributions on the CMDs 8 and 9 will differ unless the light flux isapplied along the optical axis of the objective lens 6. The fourthembodiment therefore has two light-amount correcting circuits 47 and 48of the same type used in the third embodiment (FIG. 17).

[0223] As indicated above, in the image processing apparatus accordingto the fourth embodiment of the invention, the separator lenses is areused, in place of prisms, to split the input light flux into two parts.Since the lenses is are smaller than prisms, the light-splitting sectionof the apparatus can easily be made small.

[0224] Another image processing apparatus, which is a fifth embodimentof this invention, will be described with reference to FIGS. 20 and 21.As is evident from FIG. 20, the fifth embodiment is similar to theembodiment of FIG. 18, and the same components as those of theembodiment of FIG. 18 are denoted at the same reference numerals in FIG.20 and will not be described in detail.

[0225] As has been described, in the first embodiment, the valuesinterpolated for the pixels d_(ij) of one half of the display screen(i=u+1 to u+v, j=1 to w) are interpolated, whereas the values for thepixels of the other half of the screen are the pixel signals which theCMD 8 has output. The interpolated values of the screen pixels maydeteriorated in some case, as compared to those which are the pixelsignals output by the CMD 8, and the left and right halves of the imagethe first embodiment forms may differ in resolution.

[0226] The fifth embodiment is designed to form a single image ofuniform resolution. As FIG. 21 shows, CMDs 8 and 9 (FIG. 20) are sopositioned that their imaging areas incline at the same angle to adisplay area of a display section 31 (FIG. 20). Thus, as is shown inFIG. 21, if the imaging area of the CMD 8 is inclined at angle 8 to thatof the CMD 9, the imaging areas of the CMDs 8 and 9 incline at an angleof θ/2 to the display area. In this case, the values of the screenpixels d_(ij) (i=1 to (u+v)/2, j 1 to w) defining the half display arealeft of the broken line are interpolated from the pixel signals outputby the CMD 8, whereas the values of the screen pixels d_(ij) (i=(u+u)/2to u+v, j=1 to w) defining the half display area on the right of thebroken line are interpolated from the pixel signals output by the CMD 9.

[0227] The fifth embodiment has a CMD rotating mechanism 49. Themechanism 49 rotates the CMDs 8 and 9, inclining their imaging areas atthe same angle to the display area, if the imaging areas of the CMDs 8and 9 incline to the display area when the image processing apparatus isheld with its display area extending horizontally. The angle by whichthe mechanism 49 rotates either imaging area to the display area isdetermined by the conversion factors R and S which have been detected bya displacement-detecting circuit 24. The fifth embodiment furthercomprises an additional interpolation circuit 25, which performsinterpolation on the pixel signals output by the CMD 8 to calculate thevalues of the screen pixels defining the left half display area (FIG.20).

[0228] Since the CMD rotating mechanism 49 rotates the CMDs 8 and 9, ifnecessary, thereby inclining their imaging areas at the same angle tothe display area, the image processing apparatus can form an image whichis uniform in resolution. The imaging areas of the CMDs need not beinclined at the same angle to the display area; an image can be formedwhich has a substantially uniform resolution.

[0229] It should be noted that the CMD rotating mechanism 49, whichcharacterizes the fifth embodiment, may be incorporated in the first tofourth embodiments, as well.

[0230] An image processing apparatus, which is a sixth embodiment of theinvention, will be described with reference to FIGS. 22A and 22B. As maybe understood from FIG. 22a, the sixth embodiment has components similarto those of the first embodiment shown in FIG. 6. Therefore, the samecomponents as those of the first embodiment are denoted at the samereference numerals in FIG. 22A and will not be described in detail.

[0231] As is evident from FIG. 22A, four CMDs 51, 52, 53, and 54 areprovided, each having a 1000×1000 pixel matrix. Each CMD has as manypixels as a general-purpose NTSC imaging device. Hence, the CMDs 51 to54 can be manufactured with a much higher yield than HDTV imagingdevices which have a 1920×1035 pixel matrix. As FIG. 22B shows, the CMDs51 to 54 are mounted on a half prism 1 d and juxtaposed with the CMD 51used as positional reference, such that their imaging areas overlap atregions a, b, and c.

[0232] Like any embodiment described above, the sixth embodiment has adisplacement-detecting circuit 24. The circuit 24 detects thedisplacements of the CMDs 52, 53, and 54, each in the form of conversionfactors S and R (i.e., displacement S and rotation angle R), from theimage signals representing the overlap regions a, b, and c. The threedisplacement data items, each consisting of the factors S and R, areinput to three interpolation circuits 25, respectively.

[0233] In the sixth embodiment, the half prism id is used aslight-splitting section 1. Nonetheless, the half prism id may bereplaced by two such prisms as used in the third embodiment, one whichhas a coating on a part of its output surface and consists of twoportions having different transmission coefficients. Further, each ofthe interpolation circuits 25 may have built-in coefficient memories asin the second embodiment which is shown in FIG. 13.

[0234] Another image processing apparatus, which is a seventh embodimentof the invention, will be described. The seventh embodiment is identicalto the six embodiment (FIG. 22A), except in that its light-splittingsection is of any one of the types illustrated in FIGS. 23A, 23B, 23C,and 23D and differs from that of the sixth embodiment which is a halfprism id on which four CMDs are mounted.

[0235]FIG. 23A shows the first type of the light-splitting section 1which comprises two wedge-shaped prisms 60 and 61 and a beam splitter63. The prisms 60 and 61 and the beam splitter 63 cooperate, splittingthe input light into four parts. The parts of the input light areapplied to CMDs 55, 56, 57, and 58, forming four parts of an objectimage on the imaging areas of the CMDs 55 to 58, respectively, as isillustrated in FIG. 26.

[0236]FIG. 24A is a side view of the light-splitting section 1 shown inFIG. 23A, and FIG. 24B is a plan view thereof. As clearly illustrated inFIGS. 24A and 24B, the wedge-shaped prisms 60 and 61 split the inputlight into two parts, each of which is split by the beam splitter 63into two parts. As a result, the input light is divided into four parts.The beam splitter 62 is formed of two right-angle prisms connectedtogether. As shown in FIG. 24A, a total-reflection mirror coating isapplied to the upper half of the interface between the right-angleprisms.

[0237]FIG. 23B shows the second type of the light splitting section 1which differs from the type of FIG. 23A, in that two eccentric lenses 64and 65 are used in place of the two wedge-shaped prisms 60 and 61.Unlike the prisms 60 and 61 which deflect a light flux, the eccentriclenses 64 and 65 not only deflect a light flux but also form an image.The objective lens 6 through which the input light is applied to theeccentric lenses 64 and 65 may be that type which emits an afocal flux(see FIGS. 25A nd 25B). The light-splitting section 1 of the second type(FIG. 23B) need not be positioned so precisely with respect to theobjective lens 6, owing to the use of the eccentric lenses 64 and 65.This facilitates the assembling of the image processing apparatus. Botheccentric lenses 64 and 65 are achromatic doublets, but can be lenses ofany other types.

[0238]FIG. 23C shows the third type of the light-splitting section 1which comprises four wedge-shaped prisms 66, 67, 68, and 69. Theselenses 66 to 69 are connected, side to side, forming a 2×2 matrix whichhas a concave at the center. The input light applied via the objectivelens 6 onto the 2×2 matrix is divided into four parts, i.e., anupper-left part, a lower-left part, and an upper-right part, and alower-right part. Each of the wedge-shaped prisms is an achromatic prismconsisting of two glass components which have different refractionindices. It is desirable that a telecenteric system be located at theoutput of the objective lens 6, to prevent distortion of the image whichwould otherwise occur due to the flux refraction caused by thewedge-shaped lens 66 to 69. Hence, the telecenteric system serves toaccomplish good image synthesis.

[0239]FIG. 23D shows the fourth type of the light-splitting section 1which comprises four eccentric lenses 70, 71, 72, and 73 which areconnected, side to side, forming a 2×2 matrix. It is desirable that thislight-splitting section 1 be used in combination with an objective lens6 which emits an afocal flux.

[0240] The seventh embodiment, which has a light-splitting sectioncomprising prisms or lenses, needs light-amount correcting circuits ofthe type described above.

[0241] As may be understood from the above description, the seventhembodiment is an image processing apparatus which has four solid-stateimaging devices. The imaging devices are not restricted to CMDs.Needless to say, they may be CCDs or AMIs. If CODs for use in NTSCs,which are generally used imaging devices and have 768×480 pixels each,are utilized in the seventh embodiment, the seventh embodiment will forman image of resolution as high as about 1400×800 pixels. Alternatively,four imaging devices for use in PALs, each having 820×640 pixels, may beemployed. In this case, the seventh embodiment will form an image ofhigher resolution.

[0242] An image processing apparatus, which is an eighth embodiment ofthis invention, will be described with reference to FIG. 27. Thisembodiment is identical to the first embodiment (FIG. 6), except for thefeatures which will be described below.

[0243] The seventh embodiment has a light-splitting section whichcomprises four imaging devices. According to the present invention,however, the number of imaging devices used is not limited to four atall. The eighth embodiment of the invention is characterized in that alarge number of lenses and a large number of imaging devices, that is, alens array 74 and a CMD array 75, as is clearly shown in FIG. 27. Thelenses and the CMDs have one-to-one relation, and the CMDs have theirimaging areas overlapping in part. The lens array 74 has a light shieldformed on its entire surfaces, except for the lenses. The lens array 74can be produced at low cost by means of, for example, press-processing.

[0244] The imaging devices used in the eighth embodiment are notrestricted to CMDs. Rather, they may be CCDs, MOS devices, or the like.

[0245] It will now be explained how the imaging devices are positionedin each of the fourth to eighth embodiments described above. In thefourth to eighth embodiments, the CMDs are located close to one anotherand cannot be located at such positions as shown in FIGS. 19A to 19C.Thus, they are positioned by one of various methods which will bedescribed with reference to FIGS. 28 to 32 and FIGS. 33A to 33C, andFIG. 34.

[0246]FIGS. 28 and 29 are a plan view and a sectional view,respectively, explaining the first method of positioning CMDs. In thismethod, CMDs 81 and 82 are mounted, in the form of bare chips, on aceramic substrate 80 as is shown in FIG. 28. As is best shown in FIG.29, a sectional view taken along line 29-29 in FIG. 28, the CMDs 81 and82 are set in two square recesses formed in the surface of the ceramicsubstrate 80 and fixed with adhesive 83. The rims of either squarerecess have been planed off, so that the adhesive 83 is applied insufficient quantity. The recesses are positioned and formed so preciselythat the CMDs 81 and 82 are positioned with sufficient precision whenthey are set in the recesses. The electrodes of the CMDs 81 and 82 arebonded to the electrodes formed on the ceramic substrate 80,respectively. The electrodes on the substrate 80 are, in turn, connectedto terminals 85 formed at the edges of the substrate 80 for electricallyconnecting the CMDs 81 and 82 to external components. As shown in FIG.30, the terminals 85 may protrude downward from the edges of the ceramicsubstrate 80.

[0247] The square recesses made in the surface of the substrate 80 notonly serve the purpose of positioning the CMDs 81 and 82 with requiredprecision but also they serve to provide a broad effective imaging area.The adhesive 83 is applied to the sides of each CMD as shown in FIG. 29,not to the bottom of the CMD, so that the position of the CMD may beadjusted with respect to the optical axis of the light-splittingsection. Were the adhesive 83 applied to the bottom of the CMD, the CMDmight tilt or move to assume an undesirable position with respect to theoptical axis of the light-splitting section.

[0248] Each CMD may be fastened to the ceramic substrate 80 in anotherway. As FIG. 29 shows, a hole 87 may be in the substrate 80 bored fromthe lower surface thereof, and adhesive 88 may be applied in the hole.This method of securing the CMD to the substrate 80 is advantageous intwo respects. First, it minimizes the risk that the adhesive shouldcover the light-receiving surface of the CMD. Second, much care need notbe taken to apply the adhesive 88.

[0249]FIG. 31 is a cross-sectional view, explaining the second method ofpositioning CMDs. In the second method, a ceramic substrate 80 is bondedto a prism or a quartz filter (not shown) by means of spacers 90 mountedon both edges of the substrate 80. As a result, the substrate 80 isspaced away from the prism or the filter. Hence, no load is exerted fromthe prism or filter on the bonding wires 91 formed on the substrate 80,provided that the height H of the spacers 90 is greater than that h ofthe bonding wires 91.

[0250]FIG. 32 is a plan view, explaining the third method of positioningbare CMDs. This method is to use a substrate 80 having rectangularrecesses in its surface. Bare CMD chips 82 are placed in the recesses,respectively, each abutted on one edge of the recess and therebypositioned in the horizontal direction. The chips 82, thus positioned,are fixed to the substrate 80 by using adhesive.

[0251]FIGS. 33A to 33C explain the fourth method of positioning CMDs,which is employed to manufacture the image processing apparatusaccording to the six embodiment. FIG. 33A is a side view, FIG. 33B afront view seen in the direction of arrow B in FIG. 33A, and FIG. 33C abottom view seen in the direction of arrow A in FIG. 33A. As shown inFIG. 33C, spacers 90 are mounted on a substrate 80, thereby protectingthe bonding wires 91.

[0252]FIG. 34 is a side view, explaining the fifth method of positioningCMDs. In this method, two ceramic substrate 80 are sutured to a backingmember 92 which has a right-angle L cross section. Spacers 90 aremounted on the substrates 80, and a half prism 93 is abutted on thespacers 90. Hence, the half prism 93 is secured, at its two adjoiningsides, to the ceramic substrates 80 and spaced away therefrom by theheight H of the spacers 90.

[0253] Another image processing apparatus, which is a ninth embodimentof the invention, will be described with reference to FIG. 35A. As acomparison between FIG. 17 and FIG. 35 may reveal, the ninth embodimentis similar to the third embodiment but different in animage-synthesizing circuit 121. Therefore, the same components as thoseof the third embodiment (FIG. 17) are denoted at the same referencenumerals in FIG. 35A and will not be described in detail.

[0254] The image-synthesizing circuit 121 has the structure shown inFIG. 36. The circuit 121 comprises a pixel-value converter 122 and apixel selector 123. The value f of an input pixel and the value g ofanother input pixel to be combined with the first-mentioned pixel areinput to both the converter 122 and the selector 123. The pixel selector123 selects some pixels which are located near an overlap region, inaccordance with the vector (coordinate value) X₁ representing theposition of an output pixel. The pixel-value converter 122 converts theinput values of the two pixels so as to display an image which has nodiscontinuity. More precisely, as FIG. 37 illustrates, the converter 122converts the input values in accordance with the positions the pixelsassume within an overlap region.

[0255] Alternatively, the image-synthesizing circuit 121 may have thestructure shown in FIG. 38. That is, the circuit 121 may comprise acoefficient-setting circuit 124, two multipliers 125 a and 125 b, and anadder 126. The circuit 124 sets weighting coefficients a and b for twoinput pixel values f and g. The multiplier 125 a multiplies the pixelvalue f by the weighting coefficient a, and the multiplier 125 bmultiplies the pixel value g by the weighting coefficient b. The adder126 adds the outputs of the multipliers 125 a, generating the sum,(fa+gb), which is input as an output pixel value to the frame memory 30(FIG. 35A).

[0256] The coefficient-setting circuit 124 sets the coefficients foreither pixel at a value of “1.0” if the pixel is located outside theoverlap region and at a value linearly ranging from “0.0” to “1.0” ifthe pixel is located in the overlap region. In FIG. 39, X₁ is theordinate in the direction of combining image parts, and P₂−P₁ is thelength of the overlap region.

[0257] As may be understood from FIGS. 38 and 39, the circuit 121 shownin FIG. 38 does not change the input pixel values f and g withoutchanging them if the pixels are located outside the overlap region. Ifthe pixels are located in the overlap region, the circuit 121 linearlychanges the weighting coefficients a and b, multiplies the values f andg by the coefficients a and b, respectively, obtaining fa and gb, andadds the values fa and gb, and outputs the sum (fa+ and gb) as an outputpixel value. Hence, the resultant image has no brightness discontinuitywhich would otherwise result from the difference in sensitivity betweenthe imaging devices. Also the image-synthesizing circuit 121 can reducegeometrical discontinuity, if any, that occurs in the overlap region dueto the correlation and the interpolation which thedisplacement-detecting circuit 24 and the interpolation circuit 25produce. Thus can the circuit 121 decrease, to some degree, thebrightness discontinuity and geometrical discontinuity in the vicinityof the overlap region. Once the displacement-detecting circuit 24 hasdetected the displacement, the light-amount correcting circuits 47 and48 may be removed so that the image-synthesizing circuit 121 can be madesimple, comprising only an adder as is illustrated in FIG. 35B. This isbecause, the circuit 121 no longer needs to change the coefficients aand b linearly, since the light amounts on the imaging areas of the CMDsgradually change in the overlap region as is shown in FIG. 15.

[0258] To reduce the brightness discontinuity further, the bias gains ofthe SPs (Signal Processors) 20 and 21 may be adjusted.

[0259] An image processing apparatus, which is a tenth embodiment of theinvention, will be described with reference to FIG. 40. The tenthembodiment is identical to the ninth embodiment (FIG. 35A), except thata edge-emphasizing circuit 127 is connected between the output of aninterpolation circuit 25 and an image-synthesizing circuit 121. Thecircuit 127 is designed to restore the quality of an image which hasbeen deteriorated due to the interpolation effected by the interpolationcircuit 25. The same components as those of the ninth embodiment aredenoted at the same reference numerals in FIG. 40, and only thecharacterizing features of the tenth embodiment will be described indetail.

[0260] The edge-emphasizing circuit 127 calculates a Laplacian by usingthe local operator of a digital filter or the like. For instance, thecircuit 127 calculates a Laplacian from an original image. That is:

[0261] Output image=input image−

² input image×ω where ω is a constant (see FIG. 42D),

² is a Laplace operator. The Laplace operator used here is, for example,the operators of FIGS. 42A, 42B, and 42C. Alternatively, the followingselective image-emphasizing method may be performed:

[0262] Output image=input image−h(x,y)*2 input image where h(x,y) is,for example, an operator for detecting lines forming the input image.

[0263] Another method of-emphasizing the frame is to used a high-passfilter. To be more specific, the input image data is subjected toFourier transformation and then input to the high-pass filter. Thefilter emphasizes the high-frequency component of the image data,performing inverse Fourier transformation on the input image data.

[0264] In order to emphasize the input image uniformly, theedge-emphasis may be performed after shifting each pixel of thereference image by a predetermined distance (e.g., ½ pixel width, ⅓pixel width, or the like), interpolating the pixel, and inputting thepixel to the image-synthesizing circuit 121. FIG. 41 shows amodification of the tenth embodiment (FIG. 40) in which anedge-emphasizing circuit 127 is connected to the output of animage-synthesizing circuit 121 so that the synthesized image data outputby the circuit 121 may be edge-emphasized uniformly.

[0265] An image processing apparatus, which is an eleventh embodiment ofthe invention, will be described with reference to FIGS. 43 and 44,FIGS. 45A to 45C, and FIGS. 46A and 46B. As can be understood from FIGS.43 and 45 which show the eleventh embodiment, the embodiment ischaracterized in that the displacements of CMDs 8 and 9 are detected byusing a reference image which has such a specific pattern as shown inFIG. 45A, 45B, or 45C. If the case of the image pattern of FIG. 45A, thepositions of the intersections of the crosses are measured with highprecision. In the case of the pattern of FIG. 45B, the positions of thedots are measured with high precision. In the case of the pattern imageof FIG. 45C, the positions of the intersections of the lines aremeasured with high precision.

[0266] The reference image is photographed, whereby the CMDS 8 and 9generate image data items representing a left half-image and a righthalf-image, respectively, as can be understood from FIG. 43. The dataitems representing these half-images are input to referencepattern-detecting and displacement-calculating circuits 130 and 131,respectively. The circuits 130 and 131 detect the half-images of thereference pattern and calculate the displacements (each consisting of ashift distance and a rotation angle) of the half-images, i.e., thedisplacements of the CMDs 8 and 9, from the data 10 representing thepositions of the intersections of the crosses or lines defining theimage patterns (45A, 45B, or 45C). The displacements, thus calculated,are stored into displacement memories 132 and 133. Then, thedisplacements stored in the memories 132 and 133 are processed in thesame way as in the tenth embodiment, as can be understood from FIG. 44.

[0267] Various methods can be utilized to detect the reference patterns.To detect the pattern of FIG. 45A or 45C, the vicinity of each line maybe tracked. To detect the pattern of FIG. 45B, the center of each dotmay be detected. Many patterns other than those of FIGS. 45A, 45B and45C can be used in the eleventh embodiment.

[0268] Owing to the use of a reference image, the displacements of theCMDs 8 and 9 can be detected even if the half-images have each so narrowan overlap region that any correlation cannot help detect thedisplacements of the corresponding CMD. In this respect the eleventhembodiment is advantageous.

[0269] Another image processing apparatus, which is a twelfth embodimentof the invention, will be described with reference to FIGS. 46A and 46B,FIG. 47, and FIGS. 48A and 48B. As is evident from FIG. 47, theembodiment is characterized by the use of a reference pattern filter 135through which to apply an optical image of an object to an objectivelens 6.

[0270] The reference pattern filter 135 is either the type shown in FIG.46A or the type shown in FIG. 46B. The pattern filter of FIG. 46A has areference pattern which consists of two crosses located at the upper andlower portions of the overlap region, respectively. The pattern filterof FIG. 46B has a reference pattern which consists of two dots locatedat the upper and lower portions of the overlap region, respectively. Thereference pattern of either type is read along with the input imagehalves.

[0271] As FIG. 47 shows, the twelfth embodiment has a referencepattern-detecting and displacement-calculating circuit 136 which detectsthe reference pattern from the upper and lower edge portions of theoverlap region. More specifically, the circuit 136 detects the referencepattern of FIG. 46A by tracking the vicinity of each of the linesforming the crosses, and detects the reference pattern of FIG. 46B bydetecting the center of each dot. The circuit 136 determines thedisplacements of the left and right halves of the input image from thereference pattern. Thereafter, the same sequence of operations iscarried out as in the tenth embodiment. The reference pattern filter 135is useful and effective, particularly in the case where the input imageis one reproduced from silver salt film.

[0272] The twelfth embodiment can fast determine the positional relationbetween the left and right halves of the input image. Since thereference pattern filter 135 is used, the relative positions of theimage halves can be detected more accurately than otherwise. The filter135 may be removed from the optical path of the objective lens 6,thereby modifying the system structure quite easily.

[0273] An image processing apparatus according to a thirteenthembodiment of the invention will be described with reference to FIGS.48A and 48B and FIG. 49. This embodiment is identical to the tenthembodiment (FIG. 40), except that a rotation-angle detecting circuit 120and a rotational interpolation circuit 123 are used so that three ormore image parts may be combined into a single image. The thirteenthembodiment is designed to prevent erroneous detection of the correlationamong images even if there are many images to be combined and one imageis greatly rotated with respect to another as is shown in FIG. 48A.

[0274] The rotation angle R detected by a displacement-detecting circuit24 is input to the rotation-angle detecting circuit 120. From the angleR, the circuit 120 determines whether or not the synthesized imageoutput by an image-synthesizing circuit 7 should be processed by therotational interpolation circuit 123. To be more precise, the circuit120 connects the movable contact of a selector circuit 121 to the fixedcontact A thereof if the angle R is greater than a threshold value as isshown in FIG. 48A. In this case, the synthesized image is input to therotational interpolation circuit 123. The circuit 123 rotates the imageby angle of −R as is illustrated in FIG. 48B, and then combines theimage with a third image. The resultant image, i.e., a combination ofthree images, is stored into a frame memory 30.

[0275] If the angle R is equal to or less the threshold value, therotation-angle detecting circuit 120 connects the movable contact of aselector circuit 121 to the fixed contact B thereof. In this case, thesynthesized image is stored directly into the frame memory 30.

[0276] When the thirteenth embodiment is employed to combine three ormore images into a single image, the rotation-angle detecting circuit120, the selector circuit 121, and the rotational interpolation circuit123 cooperate to prevent erroneous correlation of images, i.e.,mis-matching of images.

[0277] Another image processing apparatus, which is a fourteenthembodiment of the invention, will be described with reference to FIGS.50A and 50B and FIG. 51. The fourteenth embodiment is identical to thetenth embodiment (FIG. 40), except that a circuit 125 is used which isdesigned to detect the ends of a border line. This embodiment isutilized to combine three or more images into one image.

[0278] If there are many images to combine, the right edge of the regionover which a first image adjoins a second image may incline as shown inFIG. 50A, and an undesired portion may be formed when the second imageis combined with a third image by the process described with referenceto FIGS. 38 and 39 since the center of the adjoining region is used asthe center in said process. The fourteenth embodiment is designed toprevent the forming of such an undesired portion.

[0279] As is shown in FIG. 51, the data representing a left image issupplied to the circuit 125. The circuit 125 detects ends A and B of theright order line of the image. The coordinates values of the end A,whose y-coordinate is less than that of the end B, is input to animage-synthesizing circuit 7. The circuit 7 uses the y-coordinate of theend A, defining the right edge of the region over which the second andthe third image adjoin as shown in FIGS. 50A and 50B. Then, as FIG. 50Bshows, the circuit 7 combine the synthesized image with the next imagesuch that the point A defines the right edge of the adjoining region andthe adjoining region is positioned with its center line passing amidpoint between the point A and the left edge of the overlap region.

[0280] In the fourteenth embodiment, the circuit 125 detects the ends Aand B of the right border line of the left image, and theimage-synthesizing circuit 7 uses the y-coordinate of the end A which isless than than that of the end B, defining the right edge of theadjoining region. As a result of this, an undesired portion iseliminated from the adjoining region.

[0281] Another image processing apparatus according to a fifteenthembodiment of the present invention will be described, with reference toFIGS. 52, 53, and 54. As is shown in FIG. 53, the fifteenth embodimentcomprises 16 CMDs, a first-type synthesis section, and second-typesynthesis sections. Each CMD has a 4000×500 pixel matrix and outputsimage data showing an image overlapping the image formed by another CMDfor about 60-pixel distance. The synthesis sections combine 16 imagedata items output by these CMDs into a single image having resolution ashigh as 4000×6000 which is the resolution achieved by silver-salt film.

[0282] The first-type synthesis section has the structure shown in FIG.52. Each of the second-type synthesis sections has the structure shownin FIG. 54. Each second-type synthesis section is connected to receivetwo inputs. The first input is an image signal supplied from a CMD, andthe second input is the image data read from the frame memory 30 of thepreceding second-type synthesis section. The second input is inputdirectly to a displacement-detecting circuit 24. Each second-typesynthesis section has a circuit 125 for eliminating an undesired portionof the adjoining region of a synthesized input image. The circuit 125serves to eliminate an undesired portion from the adjoining region of asynthesized image.

[0283] As can be understood from FIG. 53, the image signals the 16 CMDshave generated are processed in 16 image-synthesizing steps. The imageprocessing apparatus according to the fifteenth embodiment can,therefore, form an image having high resolution comparable with theresolution of 4000×6000 which is accomplished by silver-salt film.

[0284] An image processing apparatus, which is a sixteenth embodiment ofthe invention, will be described with reference to FIGS. 55 and 56. Asis evident from FIG. 55, this embodiment is similar to the fifteenthembodiment (FIG. 53), comprising 16 CMDs, first-type synthesis sections,and third-type synthesis sections. The first-type synthesis sections areidentical to the first-type synthesis section incorporated in thefifteenth embodiment and shown in detail in FIG. 52. The third-typesynthesis sections are identical and have the structure illustrated inFIG. 56.

[0285] In each of the third synthesis sections, the two data items readfrom the frame memories 30 of the preceding two synthesis sections areinput to the displacement-detecting circuit 24. Each third synthesissection has a circuit 125 for eliminating an undesired portion of theadjoining region of a synthesized input image.

[0286] The sixteenth embodiment performs many image syntheses inparallel to shorten the time for forming a synthesized image. Morespecifically, it produces a synthesized image in four sequential stepsonly, whereas the fifteenth embodiment forms a synthesized image in 15sequential steps. Obviously, the sixteenth embodiment can effectimage-synthesizing faster than the fifteenth embodiment.

[0287] In the fifteenth and sixteenth embodiments, 16 CMDs each having4000×500 pixels are utilized. Nonetheless, more or less imaging devicesof having the same number of pixels or a different number of pixels maybe incorporated, if necessary, in either embodiment.

[0288] A projector, which is a seventeenth embodiment of this invention,will be described with reference to FIGS. 57 to 62. As shown in FIG. 57,the projector 126 is designed to project a plurality of images to ascreen 127, which are combined into a single image on the screen 127. Asis shown in FIG. 58, the projector 126 has a half prism 128 and threeLCDs (Liquid-Crystal Displays) 129, 130, and 131. The LCDs displayimages, which are projected onto the screen 127 and combined thereoninto a single image. As will be explained, the projector 126 can form acombined image with virtually no discontinuity even if the LCDs 129,130, and 131 are not positioned precisely.

[0289] As shown in FIG. 59, the LDCs 129, 130, 131 are mounted on thehalf prism 128. The are so positioned that the images projected fromthem will be combined on the screen 127 into a single image which hasoverlap regions. A quartz filter 132 is placed in front of thelight-emitting surface of the half prism 128. The filter 132 functionsas a low-pass filter for preventing the individual pixels of each LCDfrom being visualized on the screen 127 to degrade the quality of theprojected image.

[0290] As is shown in FIG. 58, the seventeenth embodiment has an S,Rmemory 133 for storing the displacements (i.e., a distance S and arotation angle R) of the LCD 129, 130, and 131 which are determined in aspecific method, which will be described later.

[0291] Video signals, or image data representing an image to form on thescreen 127 is stored into the frame memory 30. The image data is dividedinto three data items representing three images which the LCDs 129, 130,and 131 are to display. The three data items are input to theinterpolation circuits 134, 135, 136, respectively. The circuits 134,135, and 136 execute interpolation on the input data items in accordancewith the displacement data read from the S,R memory 133, so that thedivided images projected onto the screen 127 from the LCDs 129, 130, and131 form a single image with no discontinuity.

[0292] The interpolated data items are supplied to multipliers 137, 138,and 139, respectively. The weighting coefficient calculator 140calculates weighting coefficients in the same way as in the ninthembodiment, as has been explained with reference to FIG. 39. Theweighting coefficients are supplied to the multipliers 137, 138, and139. The multipliers 137, 148, and 139 multiply those pixel signals ofthe interpolated data items which represent the overlap regions of threeimages to be projected onto the screen 127 by the weighting coefficientssupplied from the calculator 140. The brightness of each overlap regionwill therefore be adjusted. All pixel signals output from the multiplier137 are stored into the memory 141; all pixel signals output from themultiplier 138 into the memory 142; and all pixel signals output fromthe multiplier 139 into the memory 143. The pixel signals read from thememory 141 are input to the D/A converter 144; the pixel signals readfrom the memory 142 to the D/A converter 145; and the pixel signals readfrom the memory 143 to the D/A converter 146. The D/A converters 146,144, and 145 convert the input signals to three analog image data items,which are supplied to the LCDS 129, 130, and 131. Driven by these analogdata items, the LCDS display three images, respectively. A light source147 applies light to the LCD 130, and a light source 148 applies lightto the LCDS 129 and 131. Hence, three beams bearing the images displayedby the LCDs 129, 130, and 131, 10 respectively, are applied to thescreen 127 through the half prism 128 and the quartz filter 132. As aresult, the three images are combined on the screen 127 into a singleimage.

[0293] Because of the LCDS used, the seventeenth embodiment can be aprojector which can project a high-resolution image on a screen. Sincethe interpolation circuits 134, 135, and 136 and the S,R memory 133cooperate to compensate for the displacements of the LCDs 129, 130, and131, it is unnecessary to position the LDCs with high precision. Inaddition, since the multipliers 137, 138, and 139 multiply the pixelsignals which represent the overlap regions of three images to beprojected on the screen 127 by the weighting coefficients, the overlapregions are not conspicuous. Further, the quartz filter 132 prevents theimages of the individual LCD pixels from being projected onto the screen127, increasing the quality of the image formed on the screen 127. Threeother quartz filters may be used, each for one LCD.

[0294] With reference to FIG. 60, it will now be explained how to detectthe displacement of the LCDs 129, 130, and 131.

[0295] As FIG. 60 shows, a displacement-detecting mirror 149 isinterposed between a lens 6 and the quartz filter 132. The mirror 149 isinclined so as to receive the images projected from the LCDs 129, 130,and 131 and reflect them to a CCD 150 through a focusing lens 156.Hence, three images identical to those projected onto the screen 127 canbe focused on the light-receiving surface of the CCD 150.

[0296] To detect the displacement of the LCDs 129, 130, and 131, threereference data items representing three reference images which aregreatly correlated and not displaced at all (S=R=0) are input to theinterpolation circuits 134, 135, and 136, respectively. The circuits134, 135, and 136 do not process the input data items at all, and themultipliers 141, 142, and 143 multiply these data items by a weightingcoefficient of “1.”

[0297] At first, the first data item is supplied to the LCD 129, whichdisplays the first reference image. The mirror 149 reflects the firstreference image, and the lens 156 focuses it on the CCD 150. The CCD 150converts the first reference image into analog signals, and an A/Dconverter 151 converts the analog signals to digital data. The digitaldata is stored into a memory 153 through a switch 152 whose movablecontact is connected to the fixed contact a which in turn is connectedto the memory 153.

[0298] Next, the second data item is supplied to the LCD 130, whichdisplays the second reference image. The second reference image isfocused on the CCD 150 in the same way as the first reference image. Thesecond reference image is converted into analog signals and hence todigital data, in the same way as the first reference image. In themeantime, the movable contact of the switch 152 is moved and connectedto the fixed contact b which is connected to a memory 154. As a result,the digital data representing the second reference image is stored intothe memory 154. The data items stored in the memories 153 and 154 areread to an S,R detector 155. The detector 155 detects the displacementof the second reference image with respect to the first reference image,and produces data representing the displacement. The displacement datais stored into an S,R memory 133.

[0299] Then, the third data item is supplied to the LCD 130, whichdisplays the third reference image. The third reference image is focusedon the CCD 150 in the same way as the first reference image. The thirdreference image is converted into analog signals and hence to digitaldata, in the same way as the first reference image. Meanwhile, themovable contact of the switch 152 is moved and connected to the fixedcontact a which is connected to the memory 153, and the digital datarepresenting the third reference image is stored into the memory 153.The data items stored in the memories 153 and 154 are read to the S,Rdetector 155. The detector 155 detects the displacement of the thirdreference image with respect to the second reference image, and producesdata representing the displacement. The displacement data is stored intothe S,R memory 133.

[0300] Hence, with the projector it is possible to detect thedisplacements of the LCDs 129, 130, and 131. To obtain the threereference data items, use may be made of a reference image similar tothe one used in the eleventh embodiment (FIGS. 42A, 42B, 43C).

[0301] The mirror 149, which is used to detect the displacements of theLCDs 129, 130, and 131, may be replaced by a half mirror 156 as is shownin FIG. 61. In this case, the reference image displayed by each LCD isprojected onto the screen 127, and the light reflected from the screen127 is applied to the half mirror 156, which reflects the light to theCCD 150. Alternatively, a camera may be used exclusively for detectingthe displacements of the LCDs 129, 130, and 131.

[0302] The present invention can be applied to a CRT monitor of thestructure shown in FIG. 62. As FIG. 62 shows, the CRT monitor comprisesinterpolation circuits 161 to 165, electron guns 186 to 190, a phosphorscreen 193, and a spatial filter 194. The electron guns 186 to 190 emitelectron beams to the screen 193, thereby forming parts of an image. Theinterpolation circuits 161 to 165 process the data items representingthe image parts. As a result, the image parts will be moved linearly androtated on the screen 193, compensating the displacements of theelectron guns with respect to their desired positions, and forming animage having no discontinuity. The spatial filter 194 is a low-passfilter such as a quartz filter.

[0303] Since a plurality of electron guns are used, the distance betweenthe phosphor screen 193 and the beam-emitting section is shorter than inthe case where only one electron gun is used. The electron guns 186 to190 may be replaced by, for example, lasers or a unit comprising LEDs(having a lens) and micro-machine mirrors.

[0304] The distortion of image, caused by electromagnetic deflection,may be eliminated by means of the interpolation circuits 161 to 165. Theintervals of the scanning lines, which have changed due to the imagedistortion, may be utilized to set a cutoff frequency for the spatialfilter 194. Further, when lasers are used in place of the electron guns,spatial filters may be located in front of the lasers, respectively.

[0305] A film-editing apparatus, which is an eighteenth embodiment ofthe invention and which incorporates line sensors, will be describedwith reference to FIG. 63 and FIGS. 64A to 64E.

[0306] The film-editing apparatus comprises a loading mechanism 402, alight source 403, a focusing lens 404, an imaging section 405, a drivecircuit 407, an image-synthesizing circuit 408, a display 409, a memory410, and a printer 411.

[0307] When driven by the circuit 407, the loading mechanism 402 rewindsfilm 401. The light source 403 is located opposite to the focusing lens404, for applying light to the lens 404 through the film 401. The lens404 focuses the image recorded on the film 401 on the light-receivingsurface of the imaging section 405. The section 405 converts the inputimage into image signals, which are amplified by preamplifiers 10 a, 10b, and 10 c. The amplified signals are supplied to A/D converters 14 a,14 b, and 14 c and converted thereby to digital signals. The signalprocessors 20 a, 20 b, and 20 c perform γ correction and edge emphasison the digital signals. The digital signals, thus processed, are storedinto frame memories 22 a, 22 b, and 22 c.

[0308] The image signals read from the frame memories 22 a, 22 b, and 22c are input to the image-synthesizing circuit 408. The circuit 408,which has a structure similar to that of FIG. 55, processes the inputsignals, generating three data items representing a red (R) image, agreen (G) image, and a blue (B) image. These image data items are outputto the display 409, the memory 410, and the printer 411. The imagingsection 405 has the structure shown in FIG. 64A. That is, it comprisesthree line sensors 406 a, 406 b, and 406 c. As is evident from FIG. 64C,each line sensor is equipped with an optical RGB filter.

[0309] The film-editing apparatus is characterized in that the linesensors detect images while the film 401 is fed, passing through the gapbetween the light source 403 and the focusing lens 404, and that theimages thus read from the film 401 are combined into a single image. Tobe more specific, the images A, B, and C which the line sensors 406 a,406, and 406 c receive as is shown in FIG. 64B, are combined into asingle image which corresponds to one-frame image on the film 401. Theimages A, B, and C are displaced with respect to one another since theline sensors cannot and are not positioned with precision. Nevertheless,the mutual displacement will be compensated in the film-editingapparatus by means of the technique described above.

[0310] The line sensors 406 a, 406, and 406 c are much more inexpensivethan area sensors. Hence, the film-editing apparatus can accomplishhigh-resolution photographing at a very low cost. If the film 401 is acolor one, the apparatus can easily produce color image signals. Moreline sensors may be used, arranged in staggered fashion, as is shown inFIG. 64D. In this case, the images detected by the sensors arepositioned as is illustrated in FIG. 64E.

[0311] The film-editing apparatus can be modified in various ways. Forexample, not the film 401, but the light source 403, the lens 404, andthe imaging section 405 may be moved together parallel to the film,thereby to read images from the film 401. Further, each line sensor-RGBfilter unit may be replaced by an RGB line sensor which is designed forRGB photography. Still further, the RGB filter (FIG. 64C) may bereplaced by a rotating color filter.

[0312] An image processing apparatus according to a nineteenthembodiment of the invention will be described with reference to FIGS.65A and 65B. This embodiment uses CMDs and requires no frame memorieswhatever for assisting interpolation.

[0313] The nineteenth embodiment can perform random access andnondestructive read. The random access is to read the values of a pixelat any given position. The nondestructive read is to read pixel signalsas many times as desired, without losing signal charges, up until thepixel signals are reset. Due to the nondestructive read it is possibleto use each CMD as a sort of a memory, at least for a relatively shortperiod of time.

[0314] Utilizing the random access and the nondestructive read,interpolation can be executed without using frame memories. Moreprecisely, pixel values required for achieving interpolation are read bythe random access from the CMDs which are used in place frame memories.

[0315] As FIG. 65A shows, the image processing apparatus comprises,among other components, CMD drivers 32 a and 32 b, a system controller33, and an analog interpolation section 415.

[0316] The CMD drivers 32 a and 32 b are independently controlled by thesystem controller 33. They are identical in structure, comprising anaddress generator 412, an x-decoder 413, and a y-decoder 414 as shown inFIG. 65B. The address generator 412 generates addresses, which aresupplied to the x-decoder 413 and the y-decoder 414, respectively. Inaccordance with the input addresses the decoders 413 and 414 producepulse signals representing the position of a designated pixel. The pulsesignals produced by the CMD driver 32 a are supplied to a CMD 8, whereasthe pulse signals produced by the CMD driver 32 b are supplied to a CMD9.

[0317] The analog interpolation section 415 comprises a coefficientgenerator 416, a multiplier 417, an adder 418, a sample-hold circuit419, and a switch 420. The switch 420 connects the output of thesample-hold circuit 419 to either the ground or the adder 418.

[0318] The interpolation, which is a characterizing feature of thenineteenth embodiment, will be explained. The interpolation performed inthis embodiment is similar to that one which is effected in the firstembodiment (FIG. 6). As shown in FIGS. 5A and 5B, the signalrepresenting a pixel dij (i=1 to u, j=1 to w) is read from the CMD 8,converted to a digital signal, processed, and written into a framememory 30 at a specified address thereof. In the meantime, a pixeld_(ij) (i=u+1 to u+v, j=1 to w), the four signals representing fourpixels located around a pixel d_(ij) (i=u+1 to u+v, j=1 to w) are readfrom the CMD 9 by means of random access and nondestructive read. Theanalog interpolation section 415 executes analog operation defined bythe equation (9) on the four pixel signals. The pixel signals thusprocessed are converted to digital signals, which are subjected toedge-emphasis and then written into the frame memory 30 at specifiedaddresses thereof. The same pixel signal can be repeatedly read from theCMD 9, as many times as desired, since it is not destroyed at allwhenever read from the CMD 9.

[0319] Every time a pixel value is calculated by virtue of analoginterpolation, the switch 420 connects the output of the sample-holdcircuit 419 to the ground, thereby resetting the circuit 419 to “0.”Alternatively, the switch 420 may connect the circuit 419 to the groundonly when the value for the first of the four pixels is calculated, andconnect the circuit 419 to the adder 418 when the values for the secondto fourth pixels are calculated.

[0320] The image processing apparatus shown in FIGS. 65A and 65B cancombine a plurality of images into one, without using frame memoriesequivalent to the memories 22 and 23 which are indispensable to thefirst embodiment (FIG. 6). The apparatus can, therefore, be manufacturedat low cost.

[0321] The displacements of the CMD 8 and 9 can be measured in the samemethod as in the first embodiment. The coefficients output by thecoefficient generator 416 may be those selected from several typicalcoefficient sets prepared. If so, the generator 416 can be a small-scalecircuit. The PS converter 29 may be replaced by an image-synthesizingcircuit of the type illustrated in FIG. 38.

[0322] Another image processing (image-reproducing) apparatus, which isa twentieth embodiment of this invention, will be described withreference to FIG. 66. To read images from photographic film by aplurality of line sensors at high speed, so that these images are fastcombined and recorded, it is usually necessary to shorten the exposuretime of each line sensor. To this end, the amount of light applied tothe film must be increased. The light amount can be increased by using ahigh-power light source, but such a light source has a great size andand consumes much electric power. In the twentieth embodiment, use ismade of a special illumination unit.

[0323] As shown in FIG. 66, the illumination unit comprises a lightsource 403, a concave mirror 421, a cylindrical lens 422. The source 403emits light, the mirror 421 applies the light to the cylindrical lens422. The lens 422 converts the input light into three converged beams.The beams, which are intense and have an elongated cross-section, areapplied to photographic film 401, illuminating only those three portionsof the film 401 which oppose the line sensors of the imaging section405. Hence, image data can be fast input, without using a high-power,large light source.

[0324] An image processing apparatus according to a twenty-firstembodiment of the invention will be described, with reference to FIGS.67 and 68, FIGS. 69A to 69D, FIGS. 70A to 70D, FIG. 71, FIGS. 72A and72B, FIG. 73, and FIGS. 74A and 74B.

[0325] As FIGS. 74A and 74B show, this apparatus comprises two majorsections, i.e., an imaging section A and a recording section B. Thesection A is designed to form an image of an object, and the section Bto record or store the image formed by the section A. The image signalsdata by the imaging section A are transmitted to the recording sectionB, in the form of optical signals.

[0326] In the imaging section A, the image 201 of an object is suppliedthrough an imaging lens system 202, reflected by a mirror 203 a, andfocused on a CCD 204 (i.e., an imaging device). The mirror 203 a isconnected at one edge to a shaft 203 b and can be rotate around theshaft 230 b by means of a drive mechanism (not shown)

[0327] To take the image of the object, the drive mechanismintermittently rotates the mirror 203 a in the direction of the arrowshown in FIG. 67, whereby the imaging area of the section A shifts overthe object as shown in FIGS. 69A to 69D or FIGS. 70A to 70D. As aresult, the imaging section A can photograph the object in a wide view.The drive mechanism rotates the mirror 203 a intermittently at suchtiming that any two consecutive frame images overlap at least in part.The mirror 203 a may be rotated manually, in which case the drivemechanism can be dispensed with.

[0328] The light reflected by the mirror 203 a is input to the CCD 204.The CCD 204 converts the light into an image signal, which is suppliedto an A/D converter 205. The converter 205 converts the signal intodigital image data. The data is digitized by a digitizer 206 by the knowmethod and then compressed by a data-compressing circuit 207. The datais digitized and compressed. As a result, the digital image data isreduced so much that it can be transmitted, in the form of opticalsignals, from the imaging section A to the recording section B within ashort time. However, the data may be damaged while being transmitted,due to the ambient light. To avoid such transmission errors, a circuit208 adds error-correction codes to the compressed image data byReed-Solomon method or a similar method. The image data, now containingthe error-correction codes, is modulated by a modulator 209 and thensupplied to an LED driver 210. In accordance with the input image data,the LED driver 210 drives an LED 211, which emits optical signals 212.

[0329] At the recording section B, a light-receiving diode 213 receivesthe optical signals 212 transmitted from the imaging section A. Thesignals 212 are demodulated by a demodulator 214, which produces digitalimage data. The data is input to an error-correcting circuit 215. Thecircuit 215 eliminates errors, if any, in the data, with reference tothe error-correction codes contained in the image data. The image data,thus corrected, is supplied to a data-decoding circuit 216. Thecorrected image data is temporarily stored in a frame memory A 217.

[0330] As indicated above, the mirror 203 a is intermittently rotated,thereby shifting the imaging area of the section A intermittently and,thus, photographing the object repeatedly to form a wide-view imagethereof. The imaging section A may shake during the interval between anytwo photographing steps since it it held by hand. If this happen, theresultant frame images of the object may be displaced from one anotherso much that a mere combination of them cannot make a high-resolutionimage of the object. To form a high-resolution image, the image data isread from the memory A 217 and input to a shake-correcting circuit 218.The circuit 218, which will be later described in detail, processes theimage data, reducing the displacements of the frame images, which havebeen caused by the shaking of the section A. The data output from thecircuit 218 is stored into a frame memory B 219.

[0331] The first frame image data (representing the image photographedfirst) is not processed by the shake-correcting circuit 218 and storedinto the frame memory B 219. The circuit 218 processes the second frameimage data et seq., converting these frame image data items representframe images which are connected to the first frame image. These dataitems are also stored into the frame memory B 219.

[0332] Every pixel of the regions, over which the frame images overlapone another, is represented by the average of the values of the pixelsdefining all frame images, whereby a noise-reduced, high-quality singleimage will be obtained.

[0333] The image data items read from the frame memory B 219 aresupplied to a D/A converter 220 and converted to analog image data. Theanalog data is input to a CRT monitor 221, which displays the imagerepresented by these data items. Alternatively, the image data itemsread from the memory B 219 are supplied to a printer 222, which printsthe image. Still alternatively, the image data are input to a filingdevice 223 to enrich a data base.

[0334] With reference to FIG. 68, the shake-correcting circuit 218 willnow be described in detail. Also it will be explained how the circuit213 operates to correct the displacement of, for example, the Nth frameimage, which has resulting from the shake of the imaging section A.

[0335] The shake-correcting circuit 218 comprises two main components.One is a distance-measuring section 218 a for measuring the distancesthe Nth frame image is displaced from the two adjacent frame images, the(N−1)th frame image and the (N+1)th frame image. The other is animage-moving section 218 b for moving one adjacent frame image inparallel and rotating the other adjacent frame, so that the (N−1)th, Nthand (N+1)th frame images may be connected appropriately.

[0336] The imaging area of the section A shifts over the object, whiletilting in one direction and the other, as is illustrated in FIGS. 70Ato 70D. Hence, the image of the object appears as if moving androtating. The displacement of one frame image with respect to the nextone can, therefore, be represented by a motion vector. The motion vectorchanges from a frame image to another, because it includes a componentcorresponding to the rotation of the frame image.

[0337] The distance-measuring section 218 a determines the motionvectors at two or more points in the common region of two adjacent frameimages, thereby to measure the distance and the angle the second frameimage is displaced and rotated with with respect to the second frameimage. The distance and the angle, thus measured, are supplied to theimage-moving section 218 b. In accordance with the distance and theangle, the section 218 b converts the image data item showing the secondframe image to a data item which represents a frame image assuming aproper position with respect to the first frame image. As a result, thetwo adjacent frame images are connected in a desirable manner.

[0338] It will be explained how the distance-measuring section 218 ameasures the distance and the angle the second frame image is displacedand rotated with with respect to the second frame image. First, part ofthe data item representing the (N−1)th frame image is read from theframe memory A 217 and stored into a reference memory 232. Each frameimage has a size of 16×16 pixels in this instance. To detect thepositional relation between the (N−1)th frame image and the Nth frameimage, the two frame images are correlated. To be more specific, thedata stored in the reference memory 232, which represents a portion ofthe (N−1)th frame image (hereinafter called “reference image”), iscompared with the data representing that portion of the Nth frame image(hereinafter called “comparative image”) which assumes the same positionas said portion of the (N−1)th frame image and which is larger than saidportion of the (N−1)th frame image.

[0339] Next, as shown in FIG. 71, the reference image is moved tovarious positions over the comparative image, by means of anoverlap-region position controller 240. While the reference imageremains at each position, the value of every pixel of the referenceimage is compared with the value of the corresponding pixel of thecomparative image. m The absolute values of the differences between allpixels of the reference image, on the one hand, and the correspondingpixels of the comparative image, on the other, are added together underthe control of an addition controller 241. The sum of the absolutevalues of said differences is thereby obtained.

[0340] Then, the sums of absolute difference values, which have beenobtained when the reference image stays at the various positions overthe comparative image, are compared with one another. The position atwhich said sum of absolute difference values is the minimum is therebydetermined. The displacement which the reference image at this veryposition has with respect to the comparative image is regarded as amotion vector.

[0341] The signal output by the overlap-region position controller 240and the signal produced by the adding controller 241 are input to apixel-position calculator 233. One of the pixels of the Nth frame imagestored in the frame memory A 217 is thereby designated. The value ofthis pixel is supplied to one input of a difference calculator 234.Meanwhile, the signal output by the adding controller 241 designates oneof the pixels of the (N−1)th frame image stored in the reference memory232, and the value of the pixel thus designated is supplied to the otherinput of the difference calculator 234.

[0342] The difference calculator 234 calculates the difference betweenthe input pixel values. The difference is input to an absolute valuecalculator 235, which obtains the absolute value of the difference. Theabsolute value is supplied to an adder 236. The adder 236 adds the inputabsolute value to the absolute difference value stored in a sum memory237. Ultimately, the sum memory 237 stores the sum of 256 differencesfor the 16×16 pixels stored in the reference memory 237, under thecontrol of the adding controller 241. This sum is input to a minimumvalue calculator 238 and used as a correlation signal representing thesize of the overlap region of the (N−1)th frame image and the Nth frameimage.

[0343] The overlap region of two frame images is shifted under thecontrol of the overlap-region position controller 240, and thecorrelation signal obtained while the overlap region remains at eachposition is input to the minimum value calculator 238. The calculator238 determines the position where the correlation signal has the minimummagnitude. The displacement of the Nth frame image with respect to the(N−1)th frame image is input, as a motion vector v, to a ˜xAyABcalculator 239.

[0344] Assume that the correlation between the reference image and thecomparative image is most prominent when the reference image is locatedat the position (−x, −y), as is illustrated in FIG. 71. Then, the motionvector v is (x, y). The motion vectors are accumulated in a memory (notshown), whereby the motion vector is obtained which indicates theposition the Nth frame image has with respect to the first frame image.Motion vectors of this type are obtained for at least two given points aand b in the Nth frame image. The two motion vectors are input to theoxAyAG calculator 239. The calculator 239 calculates two motion vectorsfor the points a and b, i.e., v1(x1, y1) and v2(x2, y2).

[0345] The ΔxΔyΔθ calculator 239 calculates, from the vectors vi and v2,the position at which to write the Nth frame image (now stored in theframe memory A 217) in the frame memory B 219. This position is definedby the parallel motion distances (Δx and Δy) and counterclockwiserotation angle Δθ of the Nth frame image. How the calculator 239calculates the position will be explained with reference to FIGS. 72Aand 72B.

[0346] As can be understood from FIG. 72A, a motion vector v can beconsidered one synthesized from two vectors S and r which pertain to theparallel motion and rotation of a frame image. The motion vector v isevaluated in units of one-pixel width. Nonetheless, it can be evaluatedmore minutely by interpolating a correction value from the correlationsamong the pixels, as is disclosed in Published Unexamined JapanesePatent Application 4-96405. Namely:

Vector v1=S+r

Vector v2=S−r

[0347] Therefore, the vector S and the vector r are:

Vector S=(v1−v2)/2

Vector r=(vi=v2)/2

[0348] The components of the vector S are Δx and Δy. As evident fromFIG. 72B, the value for Δθ can be given approximately as:

Δθ=arctan (|v1−v2|/d)

[0349] The distances of parallel motion and the angle of rotation can beobtained more accurately by using not only the motion vectors for thepoints a and b, but also the motion vectors for many other points.

[0350] The parallel motion distances Δx and Δy and the rotation angle Δθare input to the image-moving section 218 b. The circuit 218 b processesthe image data showing the Nth frame image in accordance with thedistances Δx and Δy and the angle Δθ8, thereby moving the Nth frameimage linearly and rotating it. The image data item showing the Nthframe image thus moved and rotated is written into the frame memory B219. It suffices to set the center of rotation of the Nth frame image atthe mid point between the points a and b. If motion vectors arecalculated for three or more points, the center of rotation may be setin accordance with the positions of those points.

[0351] Since the pixel positions are discrete, each pixel of the Nthframe image, moved and rotated, usually does not assume the sameposition as the corresponding position in the frame memory B 219. Forthis reason, instead of the signal representing the pixel, the signalrepresenting an adjacent pixel which takes the position most similar tothat position in the memory B 218 may be written at said position in thememory B 219. Alternatively, a pixel value interpolated from the valuesof some pixels which assume positions similar to that position in thememory B 219 may be stored at the position in the memory B 219. (Themethod utilizing interpolation is preferable since it may serve to forma high-quality image.)

[0352] If any pixel of the Nth frame image is identical to one pixel ofthe (N−1)th frame image, whose value is already stored in the framememory B219, its value is not written into the memory B 219. Rather, itsvalue and the value of the identical pixel are added, in a predeterminedratio, and the resultant sum is stored into the memory B 219. Thismethod helps enhance the quality of an output image. The optimal valuefor the predetermined ratio depends on how many times the same pixel iswritten into the frame memory B 219.

[0353] In the twenty-first embodiment, the imaging area of the section Acan be switched rather roughly, and a simple means such as a polygonalmirror can be used to control the optical system for switching theimaging area. Further, the imaging section A can operate well even whileheld by hand because its shake is compensated well.

[0354]FIGS. 74A and 74B illustrate how the apparatus according to thetwenty-first embodiment is used. The imaging section A can be held byhand as shown in FIG. 74A since its shake is compensated. The imagingsection A and the recording section of FIG. 74B need not be connected bya cable and can therefore be located far from each other. This isbecause, as shown in FIGS. 74A and 74B, the section A can transmitsignals to the recording section B, in the form of infared rays or radiowaves. The imaging section A can be small and light and can therefore bemanipulated easily.

[0355] An image processing apparatus, which is a twenty-secondembodiment of this invention, will be described with reference to FIG.75. FIG. 75 shows only the components which characterize thisembodiment. Except for these components, the twenty-second embodiment isidentical to the twenty-first embodiment.

[0356] The twenty-second embodiment has an optical system designedexclusively for detecting the shake of an image.

[0357] In operation, an image 265 of an object is applied through a lenssystem 266 to a mirror 267 a. The mirror 267 reflects the image to ahalf mirror 268. The half mirror 268 reflects the image and applies itto an imaging device 269 which is a line sensor. The imaging device 269converts the image into image data, which is supplied to a CRT or aprinter (neither shown) so that the image may be displayed or printed.Meanwhile, the input image is applied through the half mirror 268 and amagnifying system 270 to an imaging device 271. As a result, the imageis magnified and focused on the imaging device 271. The device 271converts the image into image data from which a shake, if any, of theimage will be detected.

[0358] Since the image focused on the imaging device 271 has beenmagnified by the magnifying system 270, the motion vectors pertaining tothe pixels forming the image can be detected in high resolution. Hence,the parallel motion distances Ax and Ay and the rotation angle Δθ, i.e.,the factors required in reconstructing the image, can be calculated moreaccurately than in the twenty-first embodiment. As a result, thereconstructed image will have higher quality. In addition, the imagingdevice 269, which is a line sensor, can read the input image at highspeed, that is, can read many pixels per unit of time.

[0359] In the twenty-first embodiment and the twenty-second embodiment,the displacement of an image with respect to the next image taken isdetected from the positional correlation between the two images. If theimages are low-contrast ones, however, the results of the correlationcalculation are inevitably great.

[0360] With reference to FIGS. 76A and 76B, an image processingapparatus will be described which can calculate the correction withsufficient accuracy and which is a twenty-third embodiment of thepresent invention.

[0361] As FIG. 76A shows, two highly correlative objects are placedabove and below an object of photography. The “highly correlative”objects have broad bands in the nyquist frequency range, such astwo-dimensional chirp waves, random-dot patterns defined by randomnumbers, white-noise amplified patterns, or dot-image patterns.Alternatively, as shown in FIG. 76B, characters and lines may be drawnon the upper and lower edge of the image of an object.

[0362] For example, an image located near a dot-image pattern is used asa reference image in calculating the correlation. In this case, theapparatus can calculate the correlation with very high accuracy.

[0363] An image processing apparatus according to a twenty-fourthembodiment will be described with reference to FIGS. 77 to 79, FIGS. 80Ato 82C, and FIGS. 81 to 83. This embodiment can increase the accuracy ofthe correlation calculation without using highly correlative patterns ofthe types utilized in the twenty-third embodiment. The embodiment isidentical to the twenty-first embodiment (FIG. 67), except that acorrelated area selector is incorporated in a shake-correcting circuit218 of the type shown in FIG. 68, so that a highly correlative area isselected. Hence, FIG. 77 shows only the components which characterizethe twenty-third embodiment.

[0364] In operation, the image data output from a frame memory A 217 isinput to a distance calculator 218 a, an image-moving section 218 b, anda correlated area selector 218 c. The circuit 218 c selects the mosthighly correlative part of the input image, and input data representinga reference image to a distance-measuring section 218 a, which will bedescribed later.

[0365] From the two input images the distance-measuring section 218 ameasures the displacement of one of the images with respect to the otherimage. The displacement, thus measured, is supplied to the circuit 218b, which moves and rotates the first image, thus positioning the firstimage such that the first image is properly connected to the secondimage.

[0366]FIG. 78 shows, in detail, the correlated area selector 218 c. Asis evident from FIG. 78, in the selector 218 c, an image is selectedfrom among, for example, n possible images a, to an and one of possibleimages b₁ to b_(n) shown in FIG. 79, in accordance with the input imagedata, and dispersion-detecting circuits 243 and 244 detect thedispersion values σa_(i) and σb_(i) for the selected images a_(i) andb_(j). The sum σi of these dispersion values is supplied to a maximumvalue calculator 245. The calculator 245 outputs value i_(max) whichrenders the sum σ maximal. The value i_(max) is supplied to a correlatedarea reading circuit 246. The circuit 246 reads two reference imagesi_(max) and i_(max), which are input to the distance-measuring section218 a. Hence, the two images compared have high contrast, and thecorrelation calculation can therefore be performed with high accuracy.

[0367] The dispersion-detecting circuits 243 and 244 can be of varioustypes. For example, they may be a high-pass filter or a band-passfilters. Alternatively, they may be a convolution filter having suchcoefficients as is shown in FIGS. 80A-80C. Further, the circuits 243 and244 may be of the type shown in FIG. 81, wherein use is made of the sumof the value differences among adjacent pixels.

[0368] An image processing apparatus, designed to form a high-resolutionimage or a wide image, has a plurality of imaging devices. The holdersholding the imaging devices may expand or contract as their temperaturechanges with the ambient temperature or with an increase and decrease ofthe heat they generate. In such an event the relative positions of thedevices will alter, making it difficult to provide a high-quality image.To prevent the devices from changing their relative positions, theholders are usually made of material having a small thermal expansioncoefficient. Generally, such material is expensive and hard to process.The manufacturing cost of the image processing apparatus is inevitablyhigh.

[0369] In the present invention, a technique may be applied in theimaging section to avoid changes in the relative positions of theimaging devices, without using material having a small thermal expansioncoefficient. Two examples of the technique will be described withreference to FIG. 82 and FIG. 83.

[0370] In the example of FIG. 82, a beam splitter 282 for splittinginput light into two parts is secured to a holder 283, which in turn isfastened to one end of a base 281. An L-shaped holder 285 holding animaging device 284 a (e.g., a CCD), and a holder 286 holding an imagingdevice 284 b are fastened to the base 281, such that the devices 284 aand 284 b are so positioned as to receive the two light beams outputfrom the beam splitter 282 and convert them into electric signals. Inother words, the imaging devices 284 a and 284 b are set in planesconjugate to that of the semitransparent mirror of the beam splitter282. At the other end of the base 281 an optical system 287 is located.A rotary filter 288 is arranged between the beam splitter 282 and theoptical system 287.

[0371] The first imaging device 284 a is spaced apart from thesemitransparent mirror of the beam splitter 282 for a distance n. Thesecond imaging device 284 b is spaced apart from the semitransparentmirror for a distance m. The distance m is equal to the distance n, thatis, m=n. The light-receiving surface of the first device 284 a is spacedin vertical direction from the top of the base 281 by a distance q. Thescrew fastening the holder 286 to the base 281 has a play p. The play pis far less than distance q, that is, p<q.

[0372] Generally, material having an extremely small thermal expansioncoefficient is chosen for the holders 285 and 286 holding the devices284 a and 284 b, respectively, in order to prevent displacement of oneimaging device with respect to the other when the holders 285 and 286experience temperature changes. Such material is expensive and, to makematters worse, has poor processibility, and should better not be used.The materials in common use have thermal expansion coefficientsdiffering over a broad range.

[0373] In the present example shown in FIG. 82, a material having alarge thermal expansion coefficient is also positively used, ultimatelyreducing the manufacturing cost of the image processing apparatus. Morespecifically, two different materials are selected for the holders 285and 286, respectively, to satisfy the following equation:

p×α=q×β

[0374] where α and β are the thermal expansion coefficients of thematerials, respectively, p is the play p of the screw, and q is thedistance q between the base 281 and the device 284 a.

[0375] Hence, even if the holders 285 and 286 undergo temperaturechanges, the distances m and n remain equal to each other, whereby theimaging devices 284 a and 284 b are maintained, each at the sameposition relative to the other as before. Stated in another way, theyare always in planes conjugate to that of the semitransparent mirror ofthe beam splitter 282.

[0376] In the example of FIG. 83, the imaging devices 284 a and 284 bare moved not only along the axis of the optical system 287, but also ina line extending at right angles to the axis of the system 287. Moreprecisely, the positions of the holders 286 and 289 and holding thefirst imaging device 284 a and the second imaging device 284 b,respectively, are reversed as compared to the first example (FIG. 82).Suppose that the holder 286 is heated and expands in the direction a,and the holder 289 holding the device 284 a is also heated and itsvertical and horizontal portions expand in the direction a and thedirection b, respectively. As a result, the device 284 b held by thisholder 286 moves in the direction a, while the imaging device 284 a heldby the holder 289 moves in the direction b. If the displacement of thedevice 284 a in the direction b is equal to that of 10 the device 284 bin the direction a, the relative positions of the devices 284 a and 284b remain unchanged. As clearly understood from the equation of p×α= andq×β, α>β. To keep the devices 284 a and 284 b at the same relativepositions, the following equation must be satisfied:

r×α=S×β

[0377] where r is the vertical distance between the base 281 and theaxis of the imaging device 284 b, and S is the horizontal distancebetween the axis of the imaging device 284 a and the axis of the screwfastening the holder 289 to the base 281.

[0378] As evident from FIG. 83, r<S, and hence α>β. Therefore, only if rand S have values which satisfy the equation of p×α=q×β, then the twofollowing equations hold simultaneously:

p×α=q×β

r×α=S×β

[0379] In other words, since the components take the positions specifiedin FIG. 83, not only the displacement of either imaging device along theaxis of the optical system 287, but also the displacement thereof in aline extending at right angles to the axis of the system 287.

[0380] In either example it is possible to prevent changes in therelative positions of the imaging devices placed in planes conjugate tothe semitransparent mirror of the beam splitter 282 merely by selectingtwo materials having different thermal expansion coefficients for theholders supporting the imaging devices 284 a and 284 b, respectively.Neither holder needs to be made of material having a small thermalexpansion coefficient, which is expensive and has but lowprocessibility.

[0381] Assume that the materials of the holders have difference thermalexpansion coefficients which are known. Then, those portions of theholders to which the devices 284 a and 284 b are attached may havelengths determined in accordance with the known thermal expansioncoefficients. In this case as well, the relative positions of thedevices can be prevented from changing even if the holders experiencetemperature changes.

[0382] According to the present invention, the components of the imagingsection need not be made of materials having a very small thermalexpansion coefficient to avoid changes in the relative positions of theimaging devices. Rather, they are made of materials having differentlarge thermal expansion coefficients. They can yet prevent changes inthe relative positions of the imaging devices, because they have thesizes as specified above and are located at the positions describedabove.

[0383] An electronic camera, which is a twenty-fifth embodiment of theinvention, will now be described with reference to FIGS. 84 and 85.

[0384] In the twenty-first embodiment shown in FIG. 67, the mirror 203 ais arranged between the imaging lens system 202 and the imaging device204 (i.e., the CCD). Hence, the wider the input image, the greater theaberration of the image, the greater the reduction in ambient light. Thetwenty-fifth embodiment, or the electronic camera is characterized inthat, as shown in FIG. 84, a mirror 203 a is provided between an objectand an imaging lens system 202.

[0385] The electronic camera comprises a CMD 204 a having 2048×256pixels which are arranged in rows and columns as is illustrated in FIG.85. The CMD 204 a has a clock pulse generator 204-1, a horizontalscanning circuit 204-2, and a vertical scanning circuit 204-3. It shouldbe noted that the rows of pixels, each consisting of 2048 pixels, extendperpendicular to the plane of FIG. 85.

[0386] The CMD 204 a is of XY-address read type. When the clock pulsegenerator 204-1 supplies read pulses to the horizontal scanning circuit204-2 and the vertical scanning circuit 204-3, pixel signals are outputfrom the signal terminal SIG.

[0387] As FIG. 84 shows, the electronic camera further comprises astroboscopic lamp 291, polarizing filters 292 and 293, a voice coil 290,a processing section 294, a shutter-release button 299, and a memorycard 297. The lamp 291 emits flashing light to illuminate an object ofphotography. The polarizing filters 292 and 293 are positioned withtheir polarizing axes crossing at right angles. The voice coil 290 isused to rotate the mirror 203 a. The processing section 294 processesthe pixel signals output by the CMD 204 a. The memory card 297 isconnected to the section 294, for storing the image data produced bythe-CMD 204 a.

[0388] The processing section 294 has the structure shown in FIG. 86. Itcomprises an A/D converter 205, a digitizer 206, an image-synthesizingcircuit 295, a data-compressing circuit 207, a data-writing circuit 296,and a controller 298. The A/D converter 205 converts the analog pixelsignals supplied from the CMD 204 a to digital pixel signals. Thedigitizer 206 converts the digital pixel signals to binary imagesignals. The circuit 295 combines the image signals into image datarepresenting a single image. The circuit 207 compresses the image databy a specific method. The circuit 296 writes the compressed image datainto the memory card 297. The controller 298 controls all othercomponents of the processing section 294, the voice coil 290, and thestroboscopic lamp 291, every time it receives a signal generated when aphotographer pushes the shutter-release button 299.

[0389] The image-synthesizing circuit 295 comprises a fame memory A 217and a shake-correcting circuit 218—both being identical to thosedescribed above.

[0390] The electronic camera takes a picture of an object when thestroboscopic lamp 291 emits flashing light while the mirror 203 a isrotating. FIG. 87A indicates the timing of driving the stroboscopic lamp291. More precisely, FIG. 87A illustrates the timing the stroboscopiclamp 291 is driven, or changes in the voltages the vertical scanningcircuit 204-3 applies to the N vertical scanning lines. FIG. 87B showspart of the waveform of a voltage applied to the Nth vertical scanningline. As evident from FIG. 87B, the voltage applied to each line is atthe lowest level to expose the CMD 204 to light, at the intermediatelevel to read a pixel signal, and at the highest level to reset a pixel.Since the exposure timing and the signal-reading timing differ from lineto line, the stroboscopic lamp 291 is driven to emit flashing light forthe vertical blanking period during which all pixels of the CMD 204 aare exposed to light.

[0391] The operation of the electronic camera shown in FIGS. 84 to 86will be explained.

[0392] When the photographer pushes the shutter-release button 299, thevoice coil 290 rotates the mirror 203 a and the stroboscopic lamp 291emits flashing light at the time shown in FIG. 87A. The light is appliedto the object through the polarizing filter 292 and is reflected fromthe object. The reflected light is applied through the polarizing filter293 to the mirror 203 a. The mirror 203 a reflects the light, which isapplied to the CMD 204 a through the imaging lens system 202. Due to theuse of the polarizing filters 292 and 293, the light applied to the CMD204 a is free of straight reflection.

[0393] The A/D converter 205 converts the pixel signals generated by theCMD 204 a to digital signals. The digitizer 206 converts the digitalsignals to binary signals, which are input to the image-synthesizingcircuit 295. The A/D converter 205 and the digitizer 206 repeat theirfunctions a predetermined number of times, whereby the circuit 295produces image data representing an image. The data-compressing circuit207 compresses the image data. The image data compressed by the circuit207 is written into the memory card 297.

[0394] Upon applying flashing light 15 times to the object, theelectronic camera can form an image of the object which has highresolution of about 2000×3000 pixel. Since the mirror 203 a is locatedbetween the object and the imaging lens system 202, the resultant imageis free of aberration, and no reduction in the ambient light occurs.Further, the two polarizing filters 292 and 293 prevent straightreflection of the light emitted from the stroboscopic lamp 291. Sincethe period for which the lamp 291 emits a beam of light is extremelyshort, the camera shakes so little, if it does at all, during theexposure period. Hence, each frame image is not displaced with respectto the next one even though the mirror 203 a continues to rotate,whereby the resultant image is sufficiently clear.

[0395] Once the image data is written into the memory card 297 which isportable, the data can easily be transferred to a printer or a personalcomputer.

[0396] Even if the mirror 203 a is rotated at uneven speed, thecontroller 298 need not control the voice coil 290 so precisely. This isbecause a shake-correcting circuit (not shown) detects the changes inthe speed and compensates for these changes.

[0397] An electronic camera, which is a twenty-sixth embodiment of theinvention, will be described with reference to FIGS. 88 and 89 and FIGS.90A and 90B. This embodiment is similar to the twenty-fifth embodimentshown in FIG. 84. The same components as those shown in FIG. 84 aredenoted at the same reference numerals in FIG. 88, and only thecharacterizing features of the twenty-sixth embodiment will be describedin detail. In the electronic camera of FIG. 84, the flashing lightemitted from the stroboscopic lamp 291 illuminates not only the objectbut also the background thereof. In other words, the light is applied tothose areas outside the view field of the camera. This is a waste oflight.

[0398] The electronic camera shown in FIG. 88 is designed to save light.To be more specific, a reflector 300 and a lens system 301 converge theflashing light from a stroboscopic lamp 291, producing a converged lightbeam. The light beam is applied to a half mirror 302 and hence to amirror 203 a. The mirror 203 a reflects the light beam to the object.The light reflected from the object is applied to the mirror 203 a. Themirror 203 a reflects the beam, which is applied to a CMD 204 a throughthe half mirror 302 and an imaging lens system 202. Thus, the light isapplied to the object, not being wasted. The half mirror 302 has apolarizing plate and can, therefore, remove positively reflectedcomponents from the light reflected from the object.

[0399] In the case where the stroboscopic lamp 291 cannot be used, themirror 203 a may be intermittently rotated with such timing as isillustrated in FIG. 89. If the mirror 203 a is rotated at one-frameintervals, however, the image data items representing frame images maymix together. In the present embodiment, the mirror 203 a is rotated attwo-frame intervals (or longer intervals) so that the signals the CMD204 a generates during each exposure period A only may be supplied tothe image-synthesizing circuit (not shown) incorporated in a processingsection 294. The signals the CMD 204 a produces during each exposureperiod B are not used at all.

[0400] Another electronic camera, which is a twenty-seventh embodimentof the invention, will be described with reference to FIGS. 90A and 90B.This camera is characterized in that, as shown in FIG. 90A, a spring303, a cam 304 a, and a connecting rod 304 b work in concert, rotating amirror 203 b intermittently.

[0401] Alternatively, as shown in FIG. 90B, a gear 312 a and a screw 312b in mesh with the gear 312 a may be used for intermittently rotatingthe mirror 203 b. In this case, a FIT (Flame Interline Transfer)-typeCCD 204 b is used instead of the CMD 204 a. The screw 312 b has ahelical groove, each turn of which consists of a flat part and a drivenpart. As the screw 312 b rotates at constant speed, the gear 312 a isperiodically rotated and stopped. The FIT-type CCD 204 b has itseven-numbered field and its odd-numbered field exposed substantially atthe same time. The time during which to expose either field can bechanged.

[0402]FIG. 90C is a chart representing the timing at which exposure isperformed, and the angle by which to rotate the mirror 203 b, in thecase where the mirror-driving mechanism shown in FIG. 90B is employed.As long as the gear 312 a stays in mesh with any flat part of thehelical groove of the screw 312 b, the mirror 203 b remains to rotatefor some time (e.g., 10 ms). It is during this time that both theeven-numbered field and the odd-numbered field are exposed to light.While the gear 312 a stays in engagement with any driven part of thehelical groove, the mirror 203 b is rotating for some time (e.g., 20ms). During this time the signals produced by the exposure of the fieldsare supplied to a processing section 224.

[0403] The gear 312 a and the screw 312 b easily transform the rotationof the shaft of a motor to the intermittent rotation of the mirror 203b. The mirror-driving mechanism of FIG. 90B makes less noise than themechanism of FIG. 90A which comprises the cam 304 a. By virtue of themechanism shown in FIG. 90B, the frame-image data items are readilyprevented from mixing together, and the illumination light is notwasted.

[0404] The imaging device incorporated in the electronic cameras of FIG.90B is the FIT-type CCD 204 b. The CCD 204 b can be replaced by a CMD,provided that the mirror 203 b is rotated at two-frame intervals orlonger intervals.

[0405] An image processing apparatus according to a twenty-eighthembodiment of this invention will be described with reference to FIG.91. This embodiment is characterized in that a TV camera is rotated totake frame images of an object, whereas the mirror 203 b isintermittently rotated for the same purpose in the twenty-seventhembodiment (FIGS. 90A, 90B, and 90C). In the twenty-eighth embodiment,too, the frame images combined into a single image.

[0406] As FIG. 91 shows, the apparatus comprises a TV camera 305 such asa CCD camera, a processing section 294′ which performs the same functionas the section 294 shown in FIG. 84, a recording medium 306 such as ahard disk, a CRT monitor 221, and a printer 222. The section 294′comprises an A/D converter 205, an image-synthesizing circuit 295, amemory 219, and a D/A converter 220. This apparatus is designed to forma gray-scale image, and the image signals output by the TV camera 305are not converted to binary ones.

[0407] Another image processing apparatus, which is a twenty-ninthembodiment of the invention, will be described with reference to FIGS.92, 93 and 94, FIGS. 95A and 95B, and FIG. 96.

[0408] As can be understood from FIG. 92, this apparatus is similar tothe apparatus of FIG. 91 and characterized in that an ultrasonicdiagnosis apparatus 307 is used in place of the TV camera 305. Thediagnosis apparatus 307 produces a convex-type ultrasonic sonic image.This image consists of a trapezoidal image of an object and background,as is illustrated in FIG. 94. The background, which is a regionineffective, must not be used in synthesizing images such as text data.More precisely, that portion of the left image, which overlaps theineffective region of the right image as is shown in FIG. 95A, is notused in image synthesis. That portion of the right image, which overlapsthe ineffective region of the left image, is not used in imagesynthesis, either.

[0409] The left and right images are combined by processing the pixelsignals defining the overlap regions of the images as is illustrated inFIG. 96, that is, in the same way as in the first embodiment.

[0410]FIG. 93 shows the imaging section of the twenty-ninth embodiment.The output of a memory A 217 is connected to a distance calculator 218 aand an image-moving circuit 218 b. The image-moving circuit 218 b isconnected to an edge-emphasizing circuit 308 designed for effectingedge-emphasis on signals deteriorated due to interpolation. The circuit308 is connected to a left-border detector 309 for detecting the leftborder of the right image, and also to an image-synthesizing circuit311. The left-border detector 309 and a memory B 219 are connected tothe image-synthesizing circuit 311.

[0411] The memory A 217 stores image data representing the left image,whereas the memory B 219 stores image data representing the right image.The image-synthesizing circuit 311 writes two image data items into thememory B 219. The first data item represents that part of the left imagewhich is on the left of the left border of the right image. The seconddata item represents that part of the right image which is on the rightborder of the left image. The circuit 311 processes the pixel signalsdefining the overlap regions of the left and right images, and writesthe processed signals into the memory B 219. The imaging section cantherefore combine convex-type ultrasonic images appropriately.

[0412] An electronic camera, which is a thirtieth embodiment of theinvention, will be described with reference to FIGS. 97A, 97B and 97Cand FIGS. 98 to 101. This camera is designed to take three images of anobject which overlap one another as shown in FIG. 97A, and to combinethe images into a panoramic image. To be more specific, each image istaken when its left edge, seen in the field of the view finder, adjoinsthe right edge of the image taken previously and displayed in the fieldof the view finder.

[0413] As shown in FIG. 97B, the field of the view finder is comprisedof displaying sections A and B. The section A is provided to display aright edge portion of the first image previously taken. The section B isused to display the second image which adjoins the right edge portion ofthe first image displayed in the section A.

[0414] In order to photograph the image 2 shown in FIG. 97A after theimage 1 shown in FIG. 97A has been taken, a photographer pans the camerauntil the left edge of the second image adjoins that part of the firstimage which is shown in the section A. Seeing the left edge of thesecond image adjoining said part of the first image displayed in thesection A, the photographer pushes the shutter-release button,photographing the image 2.

[0415] The imaging section of the thirtieth embodiment will be describedin detail, with reference to FIG. 97C. The imaging section comprises alens 321 for focusing an input optical image, a CCD 322 for convertingthe image into electric image signals, a preamplifier 323 for amplifyingthe image signals, a signal processor 324 for performing γ correction orthe like on the image 15 signals, an A/D converter 325 for convertingthe signals to digital image signals, and a color separator 326 forseparating each digital signal into a luminance signal Y and chrominancesignals Cr. and Cb.

[0416] As FIG. 97C shows, an image-adding section 327 is connected tothe output of the color separator 326 to receive the luminance signal Y.Also, a data compressor 328 is connected to the output of the colorseparator 326 to receive the luminance signal Y and the chrominancesignals Cr and Cb and compress data formed of these input signals.

[0417] The image-adding section 327 comprises an overlap region memory329, multipliers 330 and 331, a coefficient-setting circuit 332, and anadder 333. The memory 329 is provided for storing the image datarepresenting an image previously photographed. The coefficient-settingcircuit 332 is designed to produce coefficients C1 and C2 to supply tothe multipliers 330 and 331, respectively.

[0418] In operation, a luminance signal Y is supplied to theimage-adding section 327. The section 327 adds part of the image datastored in the memory 329 to the luminance signal Y. The resultant sum issupplied from the image-adding section 327 to a D/A converter 334. Thecoefficients C1 and C2 are “1” and “0,” respectively for the displayingsection A (FIG. 97B), and are “0” and “1,” respectively, for thedisplaying section B (FIG. 97A). The output of the D/A converter 334 isconnected to a view finder 335. The view finder 335 comprises aliquid-crystal display (LCD) 336 and an ocular lens 337.

[0419] The data compressor 328 compresses the input signals Y, Cr, andCb. The compressed signals are written into a memory card 339 at thesame time the photographer pushes a shutter-release button 338. Thememory card 339 can be removed from the electronic camera. Theshutter-release button 338 is a two-step switch. When the button 338 isdepressed to the first depth, the camera measures the distance betweenitself and the object and also the intensity of the input light. Whenthe button 338 is pushed to the second depth, the camera photographs theobject. A controller 340 is connected to the image-adding section 327and also to the memory card 339, for controlling the section 327 and forcontrolling the supply of write addresses to the memory card 339.

[0420] The operation of the electronic camera according to the thirtiethembodiment of the invention will now be explained.

[0421] First, the photographer holds the camera at such a position thatthe left edge of an object is placed at the center of the field of theview finder 335. He or she then pushes the shutter-release button 338 tothe first depth. The distance-measuring system and the photometer system(either not shown) operate to adjust the focal distance and the exposuretime. The CCD 322 converts the first optical image 1 into image signals,which are amplified by the preamplifier 323. The signal processor 324effects y correction or the like on the amplified image signals. The A/Dconverter 325 converts the output signals of the processor 324 todigital signals. The color separator 326 separates each digital imagesignal into a luminance signal Y and chrominance signals Cr and Cb. Thesignals Y, Cr, and Cb are input to the data compressor 328. When thephotographer further pushes the shutter-release button 338 to the seconddepth, the data compressor 328 compresses the image data representingthe first image 1, and the compressed image data is written into thememory card and stored in a prescribed storage area of the memory card339.

[0422] In the meantime, the image data representing the right part ofthe image 1 (i.e., the overlap region 1 shown in FIG. 97A) is storedinto the overlap region memory 329. The adder 333 adds this image datato the image data representing the second image 2, generating combinedimage data. The D/A converter 334 converts the combined image data toanalog image data, which is supplied to the LCD 336. The LCD 336displays the image shown in FIG. 97B. As FIG. 97B shows, displayed inthe region A is the right edge of the image 1 which is represented bythe image data stored in the overlap region memory 329; displayed in theregion B is the second image 2 which is focused on the CCD 322 atpresent. The left edge of the image 2, which overlaps the right edge ofthe image 1 cannot be seen in the field of the view finder 335.

[0423] The camera is then panned until the position where the images 1and 2 properly adjoin each other appears in the field of the view finder335. The photographer depresses the shutter-release button 338completely, or to the second depth, upon judging that the images 1 and 2are connected appropriately. The image data of the image 2 now focusedon the CCD 322 is is thereby written in a prescribed storage area of thememory card 338. Simultaneously, the right edge of the image 2, i.e.,the area 2 overlapping the third image 3, is written in the overlapregion memory 329.

[0424] Thereafter, the third image image 3 is photographed in the sameway as the first image 1 and the second image 2. As a result, the threeframe images 1, 2, and 3 are formed. Their overlap regions 1 and 2 (FIG.97A) may be displaced from the desirable positions. Such displacementcan be compensated by the image-synthesis to be described later. Thephotographer need not pan the camera with so much care as to place theoverlap region 1 or 2 at a desired position, and can therefore take manypictures within a short time.

[0425] The images 1, 2, and 3 taken by the electronic camera shown inFIG. 97C are reproduced from the memory card 339 by theimage-reproducing apparatus shown in FIG. 98. The image-reproducingapparatus comprises a data expander 341 for expanding the image dataitems read from the memory card 339, an image-synthesizing circuit 342for combining the expanded data items, a controller 343 for controllingthe read address of the card 339 and the image-synthesizing circuit 342,a filing deice 344 for storing synthesized images, a monitor 345 fordisplaying the synthesized images, and a printer 346 for printing thesynthesized images.

[0426] The image-synthesizing circuit 342 has the structure shown inFIG. 99. It comprises three frame memories 351, 352, and 353,displacement detectors 354 and 355, interpolation circuits 356 and 357,an image-synthesizing section 358, and a frame memory 364. The framememories 351, 352, and 353 store the data items representing the images1, 2, and 3, respectively. The displacement detectors 354 and 355 detectthe displacement of the overlap regions 1 and 2 from the image dataitems read from the frame memories 351, 352, and 353. The detector 354calculates the parallel displacement S1 and rotation angle Ri of thesecond image 2, with respect to the first image 1. Similarly, thedetector 355 calculates the parallel displacement S2 and rotation angleR2 of the third image 3, with respect to the second image 2. Thedisplacement S1 and the angle R1 are 15 input to the interpolationcircuit 356, and the displacement S2 and the angle R2 to theinterpolation circuit 357.

[0427] The interpolation circuit 356 interpolates the pixel signals readfrom the second frame memory 352 and representing the second image 2,thereby producing a data item showing an image appropriately adjoiningthe first image 1. The interpolation circuit 357 interpolates the pixelsignals read from the third frame memory 353 and representing the thirdimage 3, thereby producing a data item representing an image properlyadjoining the second image 2. The image data items produced by thecircuits 356 and 357 are input to the image-synthesizing section 358.

[0428] As shown in FIG. 99, the image-synthesizing section 358 comprisesmultipliers 359, 360, and 361, a coefficient-setting circuit 362, and anadder 363. The circuit 362 is designed to produce coefficients a, b, andc for the images 1, 2, and 3, respectively. The coefficients a, b, and clinearly change in the overlap regions 1 and 2 as is illustrated in FIG.100. The image-synthesizing section 358 calculates values for the pixelsignals defining the image which the image-synthesizing circuit 342 isto output. These values are stored, in the form of image data, into theframe memory 364.

[0429] The image data representing the combined image is read from theframe memory 364, and is supplied to the filing deice 344, the monitor345, and the printer 346—all incorporated in the image-reproducingapparatus shown in FIG. 98. Hence, the synthesized, panoramic image isthereby recorded by the filing device 344, displayed on the monitor 345,and printed by the printer 346.

[0430] The image-reproducing apparatus, which combines the frame imagesproduced by the electronic camera (FIG. 97C), may be built within theelectronic camera.

[0431] In the thirtieth embodiment, only the right edge of the imageprevious taken is displayed in the section A of the view-finder field,while the image being taken is displayed in the section B of theview-finder field. Instead, both images may be displayed such that theyoverlap in the display section A. To accomplish this it suffices for thephotographer to operate the coefficient-setting circuit 362, therebysetting the coefficients C1 and C2 at 0.5 for the display section A andat 1 and 0, respectively, for the display section B, and to pan thecamera such that the second image overlaps, in part, the first imagedisplayed in the section B. Thus, the photographer can take imagesoverlapping in a desired manner, at high speed.

[0432] The signals supplied to the LCD 336 are exclusively luminancesignals Y, and the images the LCD 336 can display are monochromic.Nonetheless, the LCD 335 may be replaced by a color LCD. The color LCD,if used, may display the two images in different colors so that they maybe distinguished more clearly than otherwise. Further, the image signalsread from the overlap region memory 329 may be input to an HPF(High-Pass Filter) 365 and be thereby subjected to high-pass filtering,such as a Laplacian operation, as is illustrated in FIG. 101, the twoframe images can be more easily overlapped in a desired manner.

[0433] As has been described, the thirtieth embodiment is designed totake three frame images by panning the camera and to combine them into apanoramic image. Instead, four or more frame images may be combined intoa single wider image.

[0434] Still another electronic camera, which is a thirty-firstembodiment of this invention, will now be described with reference toFIGS. 102 and 103. This electronic camera is similar to the camera (FIG.97C) according to the thirtieth embodiment of the invention. Hence, thesame components as those shown in FIG. 97C are designated at the samereference numerals in FIG. 102, and will not be described in detail.

[0435] The camera shown in FIG. 102 is characterized in three respects.First, a correlator 371 is used which finds the correlation between theimage data read from the overlap region memory 329 and the datarepresenting the image being taken, thereby to calculate thedisplacement of the image with respect to the image previously taken.Second, an arrow indicator 372 is incorporated in the view finder 335,for indicating the displacement calculated by the correlator 371. Third,an audio output device 373 is incorporated to generate a sound or aspeech informing a photographer of the direction in which the camera isbeing moved.

[0436] The arrow indicator 372 displays an arrow in the field of theview finder 335. The arrow may extend upwards, downwards, to the left,or to the right, indicating how much the image is displaced in whichdirection, with respect to, as FIG. 103 shows, the image previouslytaken. The indicator 372 has a light source 374 which emits red lightand blue light.

[0437] If the correlation the correlator 371 has calculated has a verysmall value (indicating that the two frame images do not overlap), thelight source 374 emits red light. In the case where the correlation hasbeen correctly detected, determining the displacement of the secondimage with respect to the first, then the indicator 372 displays a arrowextending in the direction the first image is displaced. The camera ismoved to bring the second image to a position where the image properlyoverlaps the first image, thus reducing the displace to substantially“0.” At this time, the light source 374 emits blue light.

[0438] Not only is an arrow displayed in the field of the view, finder335, but also the audio output device 373 gives forth an audio message,as “Pan the camera to the right!” or “Pan the camera to the left!,”instructing the photographer to pan the camera in that direction. If thedisplacement is large, the device 373 may generate a message “Pan thecamera much to the left!” or a message “Pan the camera a little to theright.” Alternatively, the arrow indicator 372 may display a blinkingarrow indicating that the second image is displaced excessively.

[0439] A thirty-second embodiment of the present invention will bedescribed with reference to FIGS. 104A and 104B. In this embodiment,nine frame images overlapping one another as shown in FIG. 104A arecombined into a large single image. The numerals shown in FIG. 104Aindicate the order in which the images are photographed. To take theimage 5, a photographer moves the camera so that the LCD of the viewfinder displays the images 2, 4 and 5 at such positions as is shown inFIG. 104B. When the upper and right edges of the image 5 appropriatelyoverlap the lower edge of the image 2 and the left edge of the image 4,respectively, the photographer depresses the shutter-release button,thereby taking the image 5.

[0440] Since the LCD displays not only a frame image located on the leftor right side of the target image, but a frame located above or belowthe target image, it is possible with the thirty-second embodiment tophotograph many frame images arranged in both the horizontal directionand the vertical direction, overlapping one another. To achieve thismulti-image photographing, the imaging section (not shown) of thisembodiment needs an overlap region memory which has a greater storagecapacity than the overlap region memory 329 used in the thirtiethembodiment (97C.)

[0441] An image processing apparatus according to a thirty-thirdembodiment of the invention will be described, with reference to FIGS.105A and 105B. This embodiment is a data-reading apparatus for readingdata from a flat original. As is shown in FIG. 105A, the imaging section376 of the apparatus is attached to a stay 374 protruding upwards from abase 376 and located above the base 376. A shutter-release button 377 ismounted on the base 376. When the button 376 is pushed, the imagingsection 375 photographs the image data of an original placed on the base376. The imaging section 375 has a view finder 378. A memory card 379 isremovably inserted into the imaging section 375.

[0442] A photographer does not move the imaging section 375 as in thethirtieth embodiment. Rather, he or she moves the original on the base376 and takes frame images of the original. The photographer pushes theshutter-release button when he or she sees the target part of theoriginal is displayed in the field of the view finder 378.

[0443] An XY stage 380 may be mounted on the base 376 as is illustratedin FIG. 105B, and the original may be placed on the XY stage 380. Inthis case, the stage 38 can be automatically moved along the X axis andthe Y axis in accordance with the displacement which the correlator 371has calculated and which the frame image being taken has with respect tothe frame image previously taken. In other words, the photographer isnot bothered to move the original to locate the image of the desiredpart of the original in the field of the view finder 378. Alternatively,a drive mechanism (not shown) may drive the stay 374 along the X axisand the Y axis in accordance with the displacement which the correlator371 has calculated.

[0444] To identify each image taken, a numeral or any ID mark may besuperimposed on the image. Further it is possible for the photographerto operate a switch on the imaging section 375, displaying, in theview-finder field, all frame images taken thus far of an original, sothat he or she may recognize what a single combined image would looklike. Still further, the CCD incorporated in the imaging section 375 maybe replaced by a line sensor.

[0445] Another image processing apparatus, which is a thirty-fourthembodiment of this invention, will be described with reference to FIGS.106 to 108 and FIGS. 109A to 109C. This embodiment is a modification ofthe film-editing apparatus shown in FIG. 63, which 15 uses photographicfilm.

[0446] The film-editing apparatus shown in FIG. 106 uses a special typeof photographic film 401. As FIG. 107 shows, the film 401 has a seriesof imaging areas 425 and two series of magnetic tracks 426 extendingalong the perforations, or along the edges of the imaging areas 425. Anaddress signal of the type shown in FIG. 108, consisting of Os and is,is recorded on each magnetic track 426. In this embodiment, the imageformed in each imaging area 425 of the film 401 is divided into threeimages 425 a, 425 b, and 425 c, as is shown in FIGS. 109A, 109B, and109C. These images 425 a, 425 b, and 425 c will be detected by animaging device (later described).

[0447] As can be understood from FIG. 106, a controller 33 controls amotor controller 407, which in turn drives an electric motor 402. Themotor 402 rotates the film take-up shaft, whereby the film 401 loaded ina film-feeding mechanism 431 is taken up around. The take-up shaft. Twomagnetic heads 427 a and 427 b are in contact with the film 401 to readthe address signals from the magnetic tracks 426 of the film 401. Alight source 403 is located near the film 401, for applyingimage-reading light to the film 401.

[0448] The optical image read from each imaging area 425 of the film 401is focused on a CMD 405 a, i.e., a solid-state imaging device, by meansof an optical system 404. 15 (The CMD 405 a is used since it can beshaped relatively freely.) The CMD 405 a converts the input opticalimage into image signals, which are amplified by a preamplifier 10. AnA/D converter 14 converts the amplified signals to digital signals,which are input to a signal processor (SP) 20. The converter 20generates three data items representing the images 425 a, 425 b, and 425c, respectively. These image data items are stored into frame memories22 a, 22 b, and 22 c, respectively.

[0449] A low-pass filter (LPF) may be connected between the preamplifier10 and the A/D converter 14, for removing noise components from theamplified image signals. Further, a FPN (Fixed Pattern Noise)-removingcircuit may be incorporated in the CMD 405 a.

[0450] Meanwhile, the address signals read by the magnetic heads 427 aand 427 b are supplied to counters 428 and 429, which count thesesignals. When the count of either counter reaches a predetermined value,the controller 33 causes the motor controller 407 to stop the motor 402,terminating the take-up of the film 401. The count values of bothcounters 428 and 429 are input to a displacement-determining circuit430. The circuit 430 determines the displacement of the film withrespect to a prescribed position, from the count values the counters 428and 429 have when the film take-up is stopped. he displacement isdefined by a rotation angle R and a parallel displacement S, which havebeen calculated in the same method as has been explained in connectionwith the first embodiment of the present invention.

[0451] The controller 33 controls the frame memories 22 a, 22 b, and 22c, reading the image data items therefrom to an image-synthesizingcircuit 408. The circuit 408 combines the input image data items inaccordance with the rotation angle R and the parallel displacement Swhich have been detected by the displacement-determining circuit 430. Asa result, the image recorded in each imaging area 425 of the film 401 isreconstructed in the same way as has been explained in conjunction withthe first embodiment of the invention.

[0452] The image data representing the image reconstructed by thecircuit 408 is input to a display 409, a data storage 410, or a printer411.

[0453] It will now be explained how the film-editing apparatus of FIG.106 performs its function.

[0454] First, the film 401 is loaded into the film-feeding mechanism 431and is taken up around the take-up shaft. In the process, the counters428 and 429 count address signals the magnetic heads 427 a and 427 bread from the magnetic tracks 426.

[0455] When the count of either counter reaches the predetermined value,the film-feeding mechanism 431 is stopped, and the magnetic heads 427 aand 427 b move relative to the film 401 to positions B, when the film401 is stopped—as is shown in FIG. 109A. Next, the light source 403applies light to the film 401, reading a first part of the imagerecorded in the imaging area 425 of the film 401. The optical system 404focuses the image, thus read, on the CMD 405 a. The CMD 405 a convertsthe input optical image into image signals, which are processed by thepreamplifier 10, the A/D converter 14, and the signal processor 20, intoan image data item representing the first part of the image. This imagedata item is written into the frame memory 22 a.

[0456] Thereafter, the magnetic heads 427 a and 427 b move relative tothe film 401 to position C, when the film 401 is stopped, and then theheads 427 a and 427 b move relative to the film 401 to position D—as isillustrated in FIG. 109b. The light source 403, the optical system 404,the CMD 405 a, the preamplifier 10, the A/D converter 14, and the signalprocessor 20 operate in the same way as described in the precedingparagraph. As a result, two image data items representing the second andthird parts of the image are stored into the frame memories 22 b and 22c, respectively.

[0457] Next, the three data items are read from the frame memories 22 a,22 b, and 22 c and supplied to the image-synthesizing circuit 408. Thecircuit 408 combines the input data items, thus reconstructing the imagerecording in the imaging area 425 of the film 401—in accordance with thedisplacement data items (each consisting 15 of R and S) produced by thedisplacement-determining circuit 430.

[0458] The three parts of image shown in FIG. 109B are those which wouldbe read from the film 401 if the film 401 stopped at desired positions.In practice, the parts of the image assume positions B′, C′, and D′shown in FIG. 109C, displaced with respect to one another. This isinevitable because the film 401 cannot stop at the desired positions dueto the inertia of the film-feeding mechanism 431.

[0459] If any image part assumes an undesirable position when the film401 is stopped, the actual count of each counter is either greater orless than the predetermined value. The difference in count is equivalentto a motion vector detected and utilized in any embodiment describedabove that incorporates correlator or correlators. Thedisplacement-determining circuit 430 can accurately calculate therotation angle R and the parallel displacement S from that difference incount, and the image-synthesizing circuit 408 can combine the imageparts with high precision.

[0460] Because of the photographic film 401 with address signalsrecorded on it, the circuit 430 can accurately calculate thedisplacements of image parts even if the image parts are low-contrastones, unlike a correlator. Supplied with the displacement calculated bythe displacement-determining circuit 430, the image-synthesizing circuit408 can reconstruct a high-resolution image from the image data outputby the CMD 405 a, thought the CMD 405 a is a relatively smallsolid-state imaging device.

[0461] Nonetheless, the displacement-determining circuit 430 mayreplaced by a correlator. In this case, the correlator calculates themotion vector from the positions which the perforations of the film 401assumes relative to the CMD 405 a.

[0462] A film-editing apparatus, which is a thirty-fifth embodiment ofthe present invention, will be described with reference to FIGS. 110 and111. This apparatus is similar to the thirty-fourth embodiment shown inFIG. 106. The same components as those shown in FIG. 106 are, therefore,designated at the same reference numerals in FIG. 110, and will not bedescribed in detail.

[0463] This apparatus is characterized in that each of three parts of animage read from a photographic film 401 is divided into three parts by ahalf mirror 433, and nine data items representing the resulting nineimage parts are combined, thereby reconstructing the original image readfrom the film 401.

[0464] In operation, the image part 425 a shown in FIG. 109B read fromthe film 401 is applied by an optical system 404 to the half mirror 433.The mirror 433 divides the input image into three, which are applied tothree CCDs 432 a, 432 b, and 432 c. The CCDs 432 a, 432 b, and 432 cconvert the input three image parts into three data items, which areinput to an image pre-synthesizing circuit 434. The circuit 434 combinesthe three data items into a single data item which represents one of thethree parts of the image read from the film 401. The circuit 434combines two other sets of three data items representing the image parts425 b and 425 c shown in FIG. 109B, thereby producing two data itemswhich represent the two other parts of the image read 25 from thephotographic film 401. The three data items produced by the imagepre-synthesizing circuit 434 are stored into three frame memories 22 a,22 b, and 22 c, respectively.

[0465] These data items are read from the frame memories 22 a, 22 b, and22 c and input to an image-synthesizing circuit 408. The circuit 408combines the three input data items in accordance with the displacementdata items R and S which a displacement-determining circuit 430 hasgenerated from the counts of counters 428 and 429, as in thethirty-fourth embodiment. A single image identical to the original imageis thereby reconstructed. Reconstructed from nine image parts, theresultant image has a resolution higher than the image reconstructed bythe thirty-fourth embodiment (FIG. 106).

[0466] Another film-editing apparatus, which is a thirty-sixthembodiment of the invention, will be described with reference to FIGS.112 and 113. This apparatus is similar to the thirty-fifth embodiment ofFIG. 110. The same components as those shown in FIG. 106 are denoted atthe same reference numerals in FIG. 111, and will not be described indetail.

[0467] The thirty-sixth embodiment is characterized in that the addresssignals recorded in the magnetic tracks 426 of photographic film 401 areused to control a film-feeding mechanism 431 such that the three partsof each frame image recorded on the film 401 are located at desiredpositions (i.e., positions A, B, and C specified in FIGS. 109A and 109B)with respect to a CMD 405 a. Hence, three address signals recorded forevery frame image.

[0468] In this embodiment, the film 401 with the address signalsrecorded on it is loaded in the film-feeding mechanism 431, and magneticheads 435 a and 435 b contacting the film 401 can be moved along themagnetic tracks of the film 401 by means of drive sections 436 a and 436b which are controlled by a controller 33.

[0469] In operation, the film 401 loaded in the film-feeding mechanism431 is taken up under the control of the controller 33. When themagnetic heads 435 a and 435 b detect the first of the three addresssignals recorded for every frame image, the mechanism 431 stops the film401. The first image part is stopped not at the desired position A (FIG.109A), but at a more forward position, inevitably because of the inertiaof the film-feeding mechanism 431. Nonetheless, the controller 33controls the drive sections 436 a and 436 b such that the drive sectionsmove the heads 435 a and 435 b to the first image part. The distancesthe heads 435 a and 435 b are moved are detected by position-determiningcircuits 437 a and 437 b, which generate signals representative of thesedistances. The signals are input to a displacement-determining circuit430. The circuit 430 calculates a rotation angle R and a paralleldisplacement S from the input signals. The three image data items, whichthe CMD 405 a produces in the same way in the thirty-fourth embodiment,are stored into three frame memories 22 a, 22 b, nd 22 c and eventuallyinput to an image-synthesizing circuit 408. The circuit 408 combines thethree data items into a single image, in accordance with the angle R anddisplacement S which have been supplied from thedisplacement-determining circuit 430.

[0470] In the thirty-fourth, thirty-fifth, and thirty-sixth embodiments,a photographic film is intermittently stopped, each time upon counting apredetermined number of address signals read from the film, and thedisplacement (i.e., a rotation angle R and a parallel displacement S) ofeach image part with respect to another image part is calculated fromthe difference between said predetermined number of address signals andthe number of address signals counted the moment the film 401 isactually stopped. The data items representing the image parts arecorrected in accordance with the displacement data (R and S) and thenare combined, thereby reconstructing an image.

[0471] In the thirty-fourth, thirty-fifth and thirty-sixth embodiments,the overlap regions of the image parts are located by various methods,not by processing the data items representing the image parts as in theconventional image processing apparatuses. These embodiments cantherefore accurately calculate the displacements of the image parts, notrequiring complex components which will raise the manufacturing cost.Further, these embodiments, though simple in structure, can position theimage parts with high precision, thereby reconstructing an originalimage, even if the image parts have low contrast and their relativeposition cannot be well determined by a correlator.

[0472] In the thirty-sixth embodiment, wherein address signals of thetype shown in FIG. 113 are used, other data pulses can be added betweenany two adjacent pulses defining the positions where to stop the film401 intermittently.

[0473] As described above, in the thirty-fourth, thirty-fifth andthirty-sixth embodiments, the overlap regions of image parts aredetected by using the positioning pulses read from the photographicfilm. These embodiments can therefore reconstruct an original image withhigh precision.

[0474] An image processing apparatus according to a thirty-seventhembodiment of the invention will be described with reference to FIGS.114 and 115, FIGS. 116A and 116B, and FIGS. 117 to 121.

[0475] In the thirty-seventh embodiment, an input optical image isapplied through an optical system 502 to a color-separating prism 503.The prism 503 is, for example, a dichroic mirror for separating theinput image into a red beam, a green beam, and a blue beam. These beamsare applied to three CCDs 503 r, 503 g, and 503 b, respectively. TheCODs 503 r, 503 g, and 503 b are driven by a CCD driver 516, and convertthe red beam, the green beam, and the blue beam into image signals. Theimage signals are input to preamplifiers 504 r, 504 g, and 504 b and arethereby amplified. The amplified signals are supplied to A/D converters505 r, 505 g, and 505 b, respectively, and are converted thereby todigital signals. The digital signals are input to signal processors (SP)506 r, 506 g, and 506 b, which perform γ correction, edge-emphasis, orthe like on the input digital signals. The signals output by the signalprocessors 506 r, 506 g, and 506 b are stored into frame memories 507 r,507 g, and 507 b.

[0476] The image signals read from the frame memories 507 r, and 507 bare input to interpolation circuits 508 r and 508 b. The circuits 508 rand 508 b interpolate each red-pixel signal and each blue-pixel signalwhich correspond to one green-pixel signal, in accordance with thecoefficients read from coefficient memories 509 r and 509 b, which willbe described later.

[0477] The interpolation circuits 508 r and 508 b are identical instructure, and only the circuit 508 r will be described in detail. AsFIG. 115 shows, the circuit 508 r comprises a data-reading circuit 521and a linear interpolation circuit 522. The circuit 521 reads the valuesof four pixels, v_(b), B_(c), v_(d), and V_(e), from the frame memory507 r in accordance with the coordinates (IC_(x), IC_(y)) read from thecoefficient memory 509 r. The linear interpolation circuit 522 comprisesmultipliers 523, 524, 525, and 526 and an adder 527. The multiplier 523multiplies the pixel value Vb by the interpolation coefficient C_(b)read from the coefficient memory 509 r; the multiplier 524 multipliesthe pixel value V_(c) by the interpolation coefficient C_(c) read fromthe coefficient memory 509 r; the multiplier 525 multiplies the pixelvalue V_(d) by the interpolation coefficient C_(d) read from thecoefficient memory 509 r; and the multiplier 526 multiplies the pixelvalue V_(e) by the interpolation coefficient C_(e) read from thecoefficient memory 509 r. The products output by the multipliers 523,524, 525, and 526 are added by the adder 527. As a result, the valueV_(a) of the red pixel is interpolated. Namely:

V _(a) =C _(b) V _(b) +C _(c) V _(c) +C _(d) V _(d) +C _(e) V _(e)  (13)

[0478] The value of the blue pixel is interpolated by the interpolationcircuit 508 b in the same way.

[0479] The red-pixel value and the blue-pixel value, thus interpolated,are input to a PS (Parallel-Serial) converter 510, along with thegreen-pixel value. The PS converter 510 combines the input pixel values,forming a color image signal, e.g., an NTSC television signal. The colorimage signal is output to a monitor 511, a printer 512, or a filingdevice 520.

[0480] The CCD driver 516, the frame memories 507 r, 507 g, and 507 b,the coefficient memories 509 r and 509 b, and the PS converter 510 arecontrolled by a system controller 517.

[0481] As shown in FIG. 114B, the apparatus comprises coefficientcalculators 513 r and 513 b. The calculator 513 r comprises a correlator514 r and a coefficient-calculating circuit 515 r. Similarly, thecalculator 513 b comprises a correlator 514 b and acoefficient-calculating circuit 515 b. For the sake of simplicity, onlythe coefficient calculator 513 r will be described.

[0482] In the coefficient calculator 513 r, the correlator 514 r detectsa parallel vector s and a rotation vector r, which are input to thecoefficient-calculating circuit 515 r. The circuit 515 r calculatescoefficients Cb, Cc, Cd, and Ce from the vectors r and s.

[0483] The displacement of an image of a color, with respect to an imageof any other color, is detected in two factors, i.e., the paralleldisplacement and angle of rotation of a given pixel of the color image.To detect the displacement this way, reference areas a₁, a₂, a₃, and a₄are set in the green image as is illustrated in FIG. 116A. These areashave centers P₁, P₂, P₃, and P₄, respectively. The reference areas arelocated symmetrically with respect to a point C, each spaced aparttherefrom by a k-pixel distance. As shown in FIG. 116B, search areas b₁,b₂, b₃, and b₄ are set in the red image and the blue image. These areasb₁ to b₄ are searched for the positions corresponding to the referenceareas a₁ to a₄. From these positions, displacement vectors V1, V2, V3,and V4 corresponding to the reference areas a₁ to a₄ are detected. Eachof these displacement vectors is defined as follows and as shown in FIG.117, by a rotation vector r and a parallel vector s measured at positionp1 with respect to the point C:

V ₁=vector s+vector r   (14a)

V ₂=vector s+vector r−90   (14b)

V ₃=vector s−vector r   (14c)

V ₄=vector s+vector+90   (14d)

[0484] where r−90 and r+90 are vectors obtained by rotating vectors r by−90° and +90°, respectively

[0485] Vector r is given:

Vector r=k tan(θ)   (15)

[0486] where θ is the angle of rotation.

[0487] From the equation (13), the vectors s and r can be represented asfollows:

Vector s=(V ₁ +V ₂ +V ₃ +V ₄)/4   (16)

Vector r=(V ₁ +V ₂ −V ₃ −V ₄)/2   (17)

[0488] Thus, the parallel displacement and the rotation angle can bedetected. The rotation angle θ is given:

θ=tan⁻¹(vector r/k)   (18)

[0489]FIG. 118 shows a correlator 514 used in the thirty-seventhembodiment. In the correlator 514, a correlator 530 determines thecorrelation between the reference area a₁ and the search area b₁.Similarly, a correlator 531 detects the correlation between thereference area a₂ and the search area b₂; a correlator 532 thecorrelation between the reference area a₃ and the search area b₃; and acorrelator 533 the correlation between the reference area a₄ and thesearch area b₄. The correlators 530, 531, 532, and 533 outputdisplacement vectors V₁, V₂, V₃, and V₄.

[0490] Various methods of determining the correlation between two areashave been proposed. Utilized in this embodiment is the method in whichthe absolute sum of the values of the pixel defining the first area iscompared with that of the values of the pixels defining the second area.

[0491] The displacement vectors V₁, V₂, V₃, and V₄ are supplied from thecorrelators 530 to 533 to an SR detector 534. The detector 543 performsthe operation of the equations (16) and (17), detecting a paralleldisplacement s and a rotation vector r. The rotation vector r is inputto a θ detector 535. The detector 535 performs the operations of theequations (15) and (18) on the rotation vector r, calculating a rotationangle θ.

[0492] The coefficient-calculating circuits 515 r and 515 b, which areidentical and designed to calculate interpolation coefficients C_(b),C_(c), C_(d), and C_(e) from the vector r and the angle θ, will bedescribed with reference to FIG. 119. Either coefficient-calculatingcircuit performs linear interpolation, obtaining the value of a pixel Afrom the known values of pixels B, C, D, and E. As is evident from FIG.119, the line BC (broken line) passing the pixel A extends at rightangles to the lines FG and DE, crossing the lines FG and DE shown inFIG. 119 at points F and G, respectively. Assume BF: FC=DG:GE=m:n, andFA:AG=p:q. Then, the value V_(f) for the pixel F, and the value V_(g)for the pixel G are:

V _(f)=(nV _(b) +mv _(c))/(m+n)   (19)

V _(g)=(nV _(d) +mV _(e))/(m+n)   (20)

[0493] Hence, V_(a) is given:

V _(a)=(qV _(f) +pV _(g))/(p+q)   (21)

[0494] Setting the inter-pixel distance at “1,” then m+n=p+q=1.Therefore, Va is calculated as follows: $\begin{matrix}\begin{matrix}{V_{a} = {{q\left( {{nV}_{b} + {m\quad V_{c}}} \right)} + {p\left( {{nV}_{d},{{+ m}\quad V_{e}}} \right)}}} \\{= {{\left( {1 - p} \right)\left( {1 - m} \right)V_{b}} + {\left( {1 - p} \right)m\quad V_{c}} + {{p\left( {1 - m} \right)}V_{d}} + {pmV}_{e}}}\end{matrix} & (22)\end{matrix}$

[0495] Comparison of the equation (22) with the equation (13) will showthat:

C _(b)=(1−P)(1−m), C _(c)=(1−p)m,

C _(d) =p(1−m), C _(e) =pm   (23)

[0496] The coordinates of the pixel A are (C_(x), C_(y)) Then, thecoordinates for the pixels B, C, D, and E can be represented by:

Pixel B=(IC _(x) , IC _(y))

Pixel C=(IC _(x)+1, IC _(y))

Pixel D=(IC _(x) , IC _(y)+1)

Pixel E=(IC _(x)+1, IC _(y)+1)   (24)

[0497] where IC_(x) is the integral part of C_(x), and IC_(Y) is theintegral part of C_(y).

[0498] Position X_(r) in the red image and position X_(b) in the blueimage, which correspond to position X_(g) in the green image areidentified as:

X _(r) =R(θ_(r))(X _(g) +S _(r))   (25)

X _(b) =R(θ_(r))(X _(g) +S _(b))   (26)

[0499] where S_(r) is the parallel vector between the red and and thegreen images, θ_(r) is the rotation angle of the red image, θ_(b) is therotation angle of the blue image, and X_(r), X_(g), _(Xb) aretwo-dimensional vectors whose elements are an x-coordinate and ay-coordinate. R(θ is given as follows:

Vector v ₁=Vector r+Vector s

Vector V ₂=−(Vector r)+vector s

|Vector r|=L tan θ  (27)

[0500] The coefficient-calculating circuits 515 r and 515 b, which areidentical, have the structure illustrated in FIG. 120. As can beunderstood from FIG. 120, a coordinates-converting circuit 536 performsthe operations of equations (25) and (26), outputting the coordinatesC_(x) and C_(y) (real numbers) for the red and blue images. Thecoordinates C_(x) and C_(y) are input to integration circuits 537 and538, respectively. The circuits 537 and 538 generate the integral partIC_(X) of C_(X) and the integral part IC_(Y) of C_(y), respectively.These integral parts IC_(X) and IC_(Y) are output from thecoefficient-calculating circuit, and subtracters 539 and 540. Thesubtracter 539 subtracts IC_(X) from C_(X) supplied from thecoordinates-converting circuit 536, generating a coefficient m(=C_(X)−IC_(X)). The subtracter 540 subtracts IC_(Y) from C_(y) suppliedfrom the circuit 536, generating a coefficient p (=C_(y)−IC_(y)). Thevalues m and p are input to a coefficient calculator 541. The calculator541 calculates interpolation coefficients C_(b), C_(c), C_(d), and C_(e)from the coefficients m and p, by performing the operation of equation(23).

[0501] The coefficient memory 509 r will be described in detail, withreference to FIG. 121, and the other coefficient memory 509 b will notbe described in detail since it is identical to the memory 509 r.

[0502] As FIG. 121 shows, the coefficient memory 509 r comprisesmemories 551, 552, 553, and 555 for storing the interpolationcoefficients C_(b), C_(c), C_(d) and C_(e) supplied from thecoefficient-calculating circuit 515 r, and memories 556 and 557 forstoring the coordinates IC_(X) and IC_(Y) supplied from the circuit 515r.

[0503] The operation of the thirty-seventh embodiment will now beexplained with reference to FIG. 122. The embodiment executes two majorsteps. The first major step is to detect coefficients by means of thecoefficient calculating sections 513 r and 513 b and store thecoefficients obtained by the sections 513 r and 513 b into thecoefficient memories 509 r and 509 b. The second major step is tophotograph an object to acquire image data. The sections 513 r and 513b, which calculate coefficients and therefore are useful in the firstmajor step, need not be used in the second major step.

[0504] The first major step will be described, with reference to FIG.114. Assume that the object 501 is a test chart which is ablack-and-white image. The red image, green image, and blue imageobtained from the black-and-white image are greatly correlated. Thedisplacement the image has with respect to the green image, and thedisplacement the blue image has with respect to the green image can,therefore, be calculated with high accuracy. It is desirable that thetest chart have many spatial frequency components so that accuratecorrelation signals may be obtained at various positions.

[0505] The test chart 501 is photographed. To be more specific, thedistance-measuring system (not shown) adjusts the focal distance of theoptical system 502, and the photometer system (not shown) adjusts theexposure time of the CCDs 503 r, 503 g, 503 b. The optical image of thetest chart 501 is applied via the system 502 to the color-separatingprism 503. The prism 503 separates the input image into a red beam, agreen beam, and a blue beam. The CCDs 503 r, 503 g, and 503 b convertthese beams into image signals. The image signals are amplified by thepreamplifiers 504 r, 504 g, and 504 b such that the white balance ismaintained. The A/D converters 505 r, 505 g, and 505 b convert theamplified signals to digital signals. The signal processors 506 r, 506g, and 506 b perform γ correction, edge-emphasis, or the like on thedigital signals. The signals output by the signal processors 506 r, 506g, and 506 b are stored into the frame memories 507 r, 507 g, and 507 b.

[0506] The image signals read from the frame memories 507 r and 507 gare input to the coefficient calculator 513 r. In the calculator 513 r,the correlator 514 r detects the reference areas a₁, a₂, a₃, and a₄ ofthe green image, and the search areas b₁, b₂, b₃, and b₄ of the redimage. The correlator 514 r also detects the parallel vector S_(r)between the red image and the green image, and a rotation angle θ_(r).The vector S_(r) and the angle θ_(r) are supplied to thecoefficient-calculating circuit 515 r. The circuit 515 r calculates thecoordinates IC_(X), IC_(y) of the red image which corresponds to thetwo-dimensional vector X_(g) of the green image, and also calculatesinterpolation coefficients C_(b), C_(c), C_(d), and C_(e). The valuesoutput by the circuit 515 r are stored at the specified addresses of thecoefficient memory 509 r. These values define the imaging area of thegreen image, over which the red image, the green image, and the blueimage overlap as is illustrated in FIG. 122. The imaging area (FIG. 122)is designated by the system controller 517 in accordance with theoutputs of the correlators 514 r and 514 b. Instead, the imaging areamay be set by a user.

[0507] Meanwhile, the image signals read from the frame memories 507 gand 507 b are input to the coefficient calculator 513 b which isidentical in structure to the coefficient calculator 513 r. Thecalculator 513 b calculates the displacement between the green image andthe blue image, the coordinates IC x, IC y of the imaging area of theblue image, and interpolation coefficients C b , C c , C d , and C e .The values output by the coefficient-calculating circuit 515 b arestored at the specified addresses of the coefficient memory 509 b.

[0508] Thus, the interpolation coefficients for the imaging area overwhich the red, green and blue images overlap are calculated andsubsequently stored in the coefficient memories 509 r and 509 b, therebycompleting the first major step of registering coefficients in thememories 509 r and 509 b.

[0509] The first major step is carried out during the manufacture of theimage processing apparatus. The coefficients are already stored in thememories 509 r and 509 b when the apparatus is delivered to a user(i.e., a photographer). Therefore, the coefficient calculators 513 r and513 b can be removed from the apparatus after the coefficients have beencalculated and registered in the memories 509 r and 509 b.

[0510] The second major step, i.e., photographing an object, will beexplained, with reference to FIG. 114, on the assumption that thecoefficient calculators 513 r and 513 b have been removed from theapparatus.

[0511] First, a photographer gets the image of an object 501 with theimaging area defined above, and pushes the shutter-release button (notshown). As a result, the CCDs 503 r, 503 g, and 503 b generate red-imagedata, green-image data, and blue-image data, respectively. These imagedata items are stored into the frame memories 507 r, 507 g, and 507 b.

[0512] Then, the system controller 517 designates coordinates of aposition of the green image, which is located in the imaging area. Thecoefficients related to the position designated, i.e., the coordinatesIC_(x) and IC_(Y) and the interpolation coefficients C_(b), C_(c),C_(d), and C_(d), are read from the coefficient memory 509 r andsupplied to the interpolation circuit 508 r. The red-image data is readfrom the frame memory 507 r in accordance with the coordinates IC_(x)and IC_(Y) and input to the interpolation circuit 508 r. The circuit 508r interpolates the value for the red pixel located at that position ofthe green image which the system controller 517 has designated.

[0513] In the meantime, the coefficients related to the positiondesignated, i.e., the coordinates IC_(x) and IC_(y) and theinterpolation coefficients C_(b), C_(c), C_(d), and C_(e), are read fromthe coefficient memory 509 b and supplied to the interpolation circuit508 b. The blue-image data is read from the frame memory 507 b inaccordance with the coordinates IC_(x) and IC_(y) and input to theinterpolation circuit 508 b. The circuit 508 b interpolates the valuefor the blue pixel located at that position of the green image which thesystem controller 517 has designated.

[0514] The value of a green pixel is supplied from the frame memory 507g to the PS converter 510, the value of the red pixel is input from theinterpolation circuit 508 r to the PS converter 510, and the value ofthe blue pixel is input from the interpolation circuit 508 b to the PSconverter 510. The converter 510 combines the three pixel values,forming a color image signal. The color image signal is output to themonitor 511, the printer 512, or the filing device 520.

[0515] As can be understood from the foregoing, the thirty-seventhembodiment can provide a three-section color camera which con form ahigh-resolution color image with no color distortion.

[0516] Since the interpolation circuits 508 r and 508 b compensate colordistortion resulting form the mutual displacement of the CCDs 503 r, 503g, and 503 b, the positions of the CODs need not be adjusted as in theconventional apparatus. That is, no registration of solid-state imagingdevices is required. Since the image signals produced by the CCDs arecorrected, the thirty-seventh embodiment can form a high-resolutioncolor image even if the CCDs are not positioned with precision. Further,the mutual displacement of a red image, a green image, and a blue imagecan be accurately detected. This is because the red image and the blueimage are compared with the green image which is greatly correlative toboth the red image and the blue image.

[0517] In the thirty-seventh embodiment, four reference areas areutilized as shown in FIGS. 116A and 116B in order to detect thedisplacement of the red image and the blue image with respect to thegreen image. Instead, only two reference areas, either the areas a₁ anda₃ or the areas a₂ and a₃, may be used for that purpose. Alternatively,more than four reference areas may be set in the green image.

[0518] Moreover, the interpolation circuits 508 r and 508 b, whichperform linear interpolation, may be replaced by circuits designed toeffect spline interpolation or SINNG interpolation.

[0519] Further, the coefficient calculators 513 r and 513 b may beconnected to the camera by means of connectors. In this case, thecalculators 513 r and 513 b can be disconnected from the camera afterthe coefficients they have calculated are written into the coefficientmemories 509 r and 509 b.

[0520] The two coefficient calculators 513 r and 513 b can be replacedby a single calculator of the same type, provided that this calculatorcan be connected alternatively to the correlators 514 r and 514 b bymeans of a changeover switch.

[0521] Another image processing apparatus, which is a thirty-eighthembodiment of the present invention, will be described with reference toFIG. 123. This embodiment is similar to the thirty-seventh embodimentshown in FIG. 114. The same components as those shown in FIG. 114 aredenoted at the same reference numerals in FIG. 123, and will not bedescribed in detail.

[0522] The thirty-eighth embodiment is characterized in that rθ memories560 and 561 are used in place of the coefficient memories 509 r and 509b. The memory 560 stores only the vector r and the angle θ output by thecorrelator 514 r, and the memory 561 stores only the vector r and theangle θ output by the correlator 514 b. The memories 560 and 561 sufficeto have a storage capacity far less than that of the memories 509 r and509 b which need to store a variety of coefficients calculated by thecoefficient-calculating circuits 515 r and 515 b. In this case, however,it is necessary for the circuits 515 r and 515 b to calculateinterpolation coefficients and coordinate data in the second major stepof taking a picture of the test chart 501.

[0523] An image processing apparatus according to a thirty-ninthembodiment of the invention will be described, with reference to FIGS.124, 125, and 126. This embodiment is identical to the thirty-seventhembodiment, except for the features shown in FIGS. 124 and 125.

[0524] The thirty-ninth embodiment is characterized in that less data isstored in each coefficient memory 509 than in the thirty-seventhembodiment and that the apparatus can yet operate at as high a speed asthe thirty-seventh embodiment.

[0525] As described above, it is possible with the thirty-seventhembodiment to interpolate a position A (C_(x), C_(y)) from sets ofcoordinates which are presented in real numbers. Since the coordinatesof the position A, thus interpolated, are real numbers, there arecountless interpolation coefficients C_(b), C_(c), C_(d), and C_(e). Inthe thirty-ninth embodiment, it it assumed that one image consists ofL×L blocks having the same size, and the interpolation coefficient forthe coordinates of the center of each block is used as interpolationcoefficient for the image block. Therefore, L₂ interpolationcoefficients are required in the thirty-ninth embodiment. Serialnumbers, or block numbers, “1” to “L₂,” are assigned to the L₂ imageblocks, respectively. The block numbers and the L₂ interpolationcoefficients are stored in a memory, in one-to-one association.

[0526] The thirty-ninth embodiment comprises a coefficient-calculatingcircuit 515 a shown in FIG. 124 and a coefficient memory 509 a shown inFIG. 125. As FIG. 124 shows, the coefficient-calculating circuit 515 ahas a block number calculator 562 which is used in place of thecoefficient calculator 541 (FIG. 120). The block number calculator 562calculates a block number N from values m and p, as follows:

N=m/(1/L)+(q/(1+L))×L+1   (28)

[0527] where 0≦m<1, 0≦p<1.

[0528] As FIG. 125 shows, the coefficient memory 509 a comrises memories563, 564, 565, 566, 567, 568, and 569. The memories 563, 564, and 565are used to store the coordinate IC_(x), the coordinate IC_(y) and thecoefficient N, respectively, which the coefficient calculator 541 hasgenerated. The memories 566, 567, 568, and 569 are provided for storinginterpolation coefficients C_(b), C_(c), C_(d), and C_(e), respectively,which are associated with the image blocks. The memories 566, 567, 568,and 569 have L² memory cells each, as compared to the memories used inthe thirty-seventh embodiment which have as many memory cells as thepixels defining the imaging area. Obviously, the storage capacity of thememories 566, 567, 568, and 569 is far smaller than is required in thethirty-seventh embodiment. The storage capacity of each coefficientmemory can be reduced since the interpolation coefficients forsymmetrical pixels are identical.

[0529] In the thirty-ninth embodiment, the interpolation circuitsprocess image signals, thereby compensating the mutual displacement ofimages formed the imaging devices. No mechanical registration of theimaging devices is therefore required. The thirty-ninth embodiment canbe applied to a low-cost color image processing apparatus which can forma high-resolution color image, even if its imaging devices are notpositioned with high precision.

[0530] Additional advantages and modifications will readily occur tothose skilled in the art. Therefore, the invention in its broaderaspects is not limited to the specific details, and representativedevices shown and described herein. Accordingly, various modificationsmay be made without departing from the spirit or scope of the generalinventive concept as defined by the appended claims and theirequivalents.

What is claimed is:
 1. An image display apparatus for displaying acontinuous image by using a plurality of images having an overlappingarea, comprising: a plurality of displaying means for displaying theimages corresponding to the plurality of image signals; interpolationarithmetic means for correcting a geometric displacement between theimages displayed by said plurality of displaying means; brightnesscorrecting means for correcting brightness of the overlapping area ofthe images displayed by said plurality of displaying means; and imagepicking-up means for picking up the images displayed by said pluralityof displaying means; wherein when displaying the images, said pluralityof displaying means use the plurality of image signals which aresubjected to displacement correction of said interpolation arithmeticmeans and brightness correction of said brightness correcting means;wherein the interpolation arithmetic means detects a positionalrelationship between the images displayed by said plurality ofdisplaying means based on the images picked-up by said image picking-upmeans, and corrects the displacement between the images obtained by saidplurality of displaying means by using a correction coefficientcalculated from the positional relationship; and wherein the imagepicking-up means picks up the images displayed by said plurality ofdisplaying means by partially using an optical system of said pluralityof displaying means.
 2. An image display apparatus for displaying acontinuous image by using a plurality of image signals which correspondto a plurality of images having an overlapping area, comprising: aplurality of displaying means for displaying the images corresponding tothe plurality of image signals; interpolation arithmetic means forcorrecting a geometric displacement between the images displayed by saidplurality of displaying means; brightness correcting means forcorrecting brightness of the overlapping area of the images displayed bysaid plurality of displaying means; image picking-up means for pickingup the images displayed by said plurality of displaying means; a framememory for storing a total image signal as total image data whichcorresponds to a total image required to be displayed by said pluralityof displaying means; and a plurality of image memories for storing imagesignals as image data which respectively correspond to the images to bedisplayed by said plurality of displaying means; wherein when displayingthe images, said plurality of displaying means use the plurality ofimage signals which are subjected to displacement correction of saidinterpolation arithmetic means and brightness correction of saidbrightness correcting means; wherein the interpolation arithmetic meansdetects a positional relationship between the images displayed by saidplurality of displaying means based on the images picked-up by saidimage picking-up means, and corrects the displacement between the imagesobtained by said plurality of displaying means by using a correctioncoefficient calculated from the positional relationship; and whereindivided image signals are stored as divided image data in said pluralityof image memories, after the divided image signals are read out from theframe memory storing the total image signal, on the basis of apositional relationship between the images displayed by said pluralityof displaying means, the positional relationship being determined basedon the images picked-up by said image picking-up means.
 3. An imagedisplay apparatus for displaying a plurality of image signals producedfrom a plurality of image signals which have overlap areas such that theimages are combined into a single continuous image, said image displayapparatus comprising: an interpolation calculating means for performinginterpolation on respective ones of the plurality of image signals whichproduce the overlap areas, whereby any adjacent two of the plurality ofimages are matched to overlap each other; a brightness converting meansfor correcting in brightness the image signals producing the overlapareas, whereby joints between the plurality of images are madeinconspicuous; a plurality of memories for storing, as image data, theplurality of image signals after the image signals producing the overlapareas are interpolated by the interpolation calculating means andcorrected by the brightness converting means; a plurality of displayingmeans for reading the image data stored in the plurality of memories,and for displaying the plurality of images as said single continuousimage; and an image-pickup means including an image-pickup device forpicking up image signals corresponding to an image displayed by theplurality of display means; wherein the interpolation calculating meansperforms the interpolation on the respective ones of the plurality ofimage signals by using a coefficient calculated based on positionalrelations of the plurality of display means, which are calculated fromthe image signals corresponding to the image displayed by the pluralityof display means which are picked up by the image-pickup means.
 4. Animage display apparatus according to claim 3, further comprising animage dividing means for dividing an image desired to be displayed intothe plurality of image signals.
 5. An image display apparatus accordingto claim 3, wherein said brightness converting means multiplies aweighting coefficient from a weighting coefficient calculator by using amultiplier.
 6. An image display apparatus according to claim 3, whereinsaid image-pickup means shares a part of an optical system of thedisplaying means.
 7. An image display apparatus according to claim 3,wherein said image-pickup means comprises a camera specifically fordetecting displacement, provided independently of the image displayingapparatus, and said camera performs an image pickup operation in one ofan entire projection area and a part of the projection area.
 8. An imagedisplay apparatus according to claim 3, wherein said image-pickup meanspicks up a subject which has a good correlation, and detectsdisplacement.
 9. An image display apparatus according to claim 3,wherein the image-pickup means operates to pick up a reference imagedisplayed by the displaying means, the interpolation calculating meansdoes not perform interpolation and the brightness converting means doesnot perform conversion.
 10. An image display apparatus for dividing andprocessing an image to be displayed on a screen, comprising: an imagedividing means for dividing the image into a plurality of image signals;an interpolation calculating means for correcting displacement betweenthe plurality of image signals; a brightness converting means forcorrecting a brightness between the plurality of image signals; aplurality of displaying means for displaying the plurality of imagesignals corrected by the interpolation calculating means and brightnessconverting means; and an image-pickup means including an image-pickupdevice for picking up an image displayed by the displaying means,wherein said interpolation calculating means obtains a plurality ofpositional relations of the image displayed by the displaying means fromthe image signals picked up by the image-pickup means, and interpolatesdisplacement data using a displacement coefficient calculated inaccordance with the positional relations.